2024-03-29T05:12:56.999 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T05:12:56.999 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi/7628612 --name teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} -- /home/teuthworker/archive/teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi/7628612/orig.config.yaml 2024-03-29T05:12:57.022 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T05:12:57.110 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi/7628612 branch: squid description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: ceph-qa@ceph.com first_in_suite: false job_id: '7628612' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba workunit: branch: squid sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5276 sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba sleep_before_teardown: 0 subset: 24201/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba targets: smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL5LjwnX1CVfZXJX4VTbgv2L4/bti+oveyIlQz+3mHCwvt/qq8l19OdTSDuMkMpl8bKLAYTRDBRvSmfI9cuHMyk= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-28_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T05:12:57.111 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa; will attempt to use it 2024-03-29T05:12:57.111 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks 2024-03-29T05:12:57.112 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T05:12:57.113 INFO:teuthology.task.internal:Checking packages... 2024-03-29T05:12:57.139 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8da23624bc17fe0b1f797227e72a0db0ce7821ba' 2024-03-29T05:12:57.139 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T05:12:57.139 INFO:teuthology.packaging:ref: None 2024-03-29T05:12:57.139 INFO:teuthology.packaging:tag: None 2024-03-29T05:12:57.139 INFO:teuthology.packaging:branch: squid 2024-03-29T05:12:57.139 INFO:teuthology.packaging:sha1: 8da23624bc17fe0b1f797227e72a0db0ce7821ba 2024-03-29T05:12:57.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-03-29T05:12:57.296 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1959-g8da23624-1jammy 2024-03-29T05:12:57.298 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T05:12:57.304 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T05:12:57.304 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T05:12:57.338 INFO:teuthology.task.internal:Saving configuration 2024-03-29T05:12:57.349 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T05:12:57.355 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T05:12:57.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi/7628612', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 05:07:20.671585', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHpBv1Fgwix0udV5erfe5IrNK1LvQasAYmoYvLcnYw9K'} 2024-03-29T05:12:57.381 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T05:12:57.387 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-29T05:12:57.387 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T05:12:57.435 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f428b5c1280>, signals=[15]) 2024-03-29T05:12:57.436 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T05:12:57.442 INFO:teuthology.task.internal:Opening connections... 2024-03-29T05:12:57.442 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-03-29T05:12:57.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T05:12:57.530 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T05:12:57.537 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-03-29T05:12:57.647 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-03-29T05:12:57.647 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-03-29T05:12:57.694 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-29T05:12:57.694 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2024-03-29T05:12:57.694 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="22.04" 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=jammy 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-29T05:12:57.695 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=jammy 2024-03-29T05:12:57.696 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-03-29T05:12:57.723 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T05:12:57.735 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T05:12:57.742 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T05:12:57.743 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-03-29T05:12:57.748 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T05:12:57.754 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T05:12:57.754 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-03-29T05:12:57.799 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T05:12:57.833 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T05:12:57.851 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T05:12:57.851 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T05:12:57.852 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T05:12:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T05:12:57.857 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-29T05:12:57.906 INFO:teuthology.orchestra.run.smithi123.stdout:5.15.0-56-generic 2024-03-29T05:12:57.907 INFO:teuthology.task.kernel:Running kernel on smithi123: 5.15.0-56-generic 2024-03-29T05:12:57.907 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2024-03-29T05:12:58.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-03-29T05:12:58.164 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-29T05:12:58.187 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-29T05:12:58.262 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-29T05:12:58.277 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-29T05:12:58.396 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-03-29T05:12:58.486 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,518 kB] 2024-03-29T05:12:58.522 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-03-29T05:12:58.527 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-29T05:12:58.528 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-03-29T05:12:58.543 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-29T05:12:58.544 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-03-29T05:12:58.546 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-29T05:12:58.547 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-03-29T05:12:58.554 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-03-29T05:12:58.587 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-03-29T05:12:58.589 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-29T05:12:58.589 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-03-29T05:12:58.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-03-29T05:12:58.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-03-29T05:12:58.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-29T05:12:58.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-29T05:12:58.591 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-29T05:12:58.591 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-29T05:12:58.608 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-29T05:12:58.628 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-29T05:12:58.628 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-29T05:12:58.628 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-29T05:12:58.629 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-29T05:12:58.698 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-03-29T05:12:59.101 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-03-29T05:12:59.120 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-03-29T05:12:59.129 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-29T05:12:59.130 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-03-29T05:12:59.222 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-29T05:12:59.223 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-03-29T05:12:59.234 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-29T05:12:59.234 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-03-29T05:12:59.264 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-29T05:12:59.443 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-03-29T05:12:59.851 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-29T05:12:59.857 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-29T05:12:59.871 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-29T05:12:59.871 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-29T05:12:59.873 INFO:teuthology.orchestra.run.smithi123.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-29T05:13:04.438 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 12.6 MB in 2s (7,083 kB/s) 2024-03-29T05:13:05.766 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-03-29T05:13:05.783 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-29T05:13:05.836 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-03-29T05:13:06.031 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-03-29T05:13:06.032 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-03-29T05:13:06.186 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-03-29T05:13:06.186 INFO:teuthology.orchestra.run.smithi123.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-29T05:13:06.186 INFO:teuthology.orchestra.run.smithi123.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-29T05:13:06.186 INFO:teuthology.orchestra.run.smithi123.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-29T05:13:06.186 INFO:teuthology.orchestra.run.smithi123.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-29T05:13:06.187 INFO:teuthology.orchestra.run.smithi123.stdout: python2.7-minimal 2024-03-29T05:13:06.187 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-29T05:13:06.187 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2024-03-29T05:13:06.187 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-29T05:13:06.187 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-29T05:13:06.188 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-29T05:13:06.189 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2024-03-29T05:13:06.189 INFO:teuthology.orchestra.run.smithi123.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-29T05:13:06.189 INFO:teuthology.orchestra.run.smithi123.stdout:Recommended packages: 2024-03-29T05:13:06.189 INFO:teuthology.orchestra.run.smithi123.stdout: thermald 2024-03-29T05:13:06.232 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2024-03-29T05:13:06.232 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-29T05:13:06.232 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-29T05:13:06.232 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-29T05:13:06.233 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages will be upgraded: 2024-03-29T05:13:06.234 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-29T05:13:06.413 INFO:teuthology.orchestra.run.smithi123.stdout:3 upgraded, 5 newly installed, 0 to remove and 311 not upgraded. 2024-03-29T05:13:06.414 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 113 MB of archives. 2024-03-29T05:13:06.414 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-29T05:13:06.414 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-29T05:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-29T05:13:08.190 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-29T05:13:10.410 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-29T05:13:10.411 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-29T05:13:10.411 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-29T05:13:11.189 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-29T05:13:11.259 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-29T05:13:11.663 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 113 MB in 5s (22.5 MB/s) 2024-03-29T05:13:11.855 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-29T05:13:13.824 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-29T05:13:13.828 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-29T05:13:13.910 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:16.446 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-29T05:13:16.467 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-29T05:13:16.556 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-29T05:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-29T05:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:23.279 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-29T05:13:23.398 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-29T05:13:23.723 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-29T05:13:23.877 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-29T05:13:24.171 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-29T05:13:24.193 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-29T05:13:24.237 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-29T05:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-29T05:13:29.803 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-29T05:13:29.840 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:33.036 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-29T05:13:33.158 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-29T05:13:33.525 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-29T05:13:34.375 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:34.490 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-29T05:13:34.597 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:35.929 INFO:teuthology.orchestra.run.smithi123.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-29T05:13:35.929 INFO:teuthology.orchestra.run.smithi123.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-29T05:13:35.929 INFO:teuthology.orchestra.run.smithi123.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-29T05:13:35.929 INFO:teuthology.orchestra.run.smithi123.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-29T05:13:36.087 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:37.513 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:38.922 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-29T05:13:39.040 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-29T05:13:39.158 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-29T05:13:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-29T05:13:39.216 INFO:teuthology.orchestra.run.smithi123.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-29T05:13:53.314 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-29T05:13:53.315 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub' 2024-03-29T05:13:53.332 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-29T05:13:53.350 INFO:teuthology.orchestra.run.smithi123.stdout:Generating grub configuration file ... 2024-03-29T05:13:53.566 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-29T05:13:53.592 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-29T05:13:53.881 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-29T05:13:53.886 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-29T05:13:53.956 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-29T05:13:53.961 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-29T05:13:54.109 INFO:teuthology.orchestra.run.smithi123.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-29T05:13:54.109 INFO:teuthology.orchestra.run.smithi123.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-29T05:13:54.109 INFO:teuthology.orchestra.run.smithi123.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-29T05:13:54.146 INFO:teuthology.orchestra.run.smithi123.stdout:done 2024-03-29T05:13:54.579 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2024-03-29T05:13:54.603 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2024-03-29T05:13:54.603 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 21 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 5.15.0.101.98 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2024-03-29T05:13:54.604 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2024-03-29T05:13:54.605 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-29T05:13:54.605 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T05:13:54.605 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T05:13:54.605 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T05:13:54.605 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T05:13:54.605 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2024-03-29T05:13:54.805 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-03-29T05:13:55.033 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-29T05:13:55.033 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-29T05:13:55.043 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-29T05:13:55.063 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-29T05:13:56.389 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-03-29T05:13:56.407 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-29T05:13:56.467 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-03-29T05:13:56.664 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-03-29T05:13:56.665 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-03-29T05:13:56.837 INFO:teuthology.orchestra.run.smithi123.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-29T05:13:56.837 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-03-29T05:13:56.837 INFO:teuthology.orchestra.run.smithi123.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-29T05:13:56.838 INFO:teuthology.orchestra.run.smithi123.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-29T05:13:56.838 INFO:teuthology.orchestra.run.smithi123.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-29T05:13:56.838 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-29T05:13:56.838 INFO:teuthology.orchestra.run.smithi123.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-29T05:13:56.838 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-29T05:13:56.878 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 0 newly installed, 0 to remove and 311 not upgraded. 2024-03-29T05:13:56.880 DEBUG:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2024-03-29T05:13:56.896 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2024-03-29T05:13:56.896 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2024-03-29T05:13:56.896 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2024-03-29T05:13:56.896 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 21 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 5.15.0.101.98 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2024-03-29T05:13:56.897 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2024-03-29T05:13:56.898 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-03-29T05:13:56.945 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.frf9ahbEtz 2024-03-29T05:13:56.945 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub/grub.cfg /tmp/tmp.frf9ahbEtz 2024-03-29T05:13:57.001 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.frf9ahbEtz 2024-03-29T05:13:57.115 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.frf9ahbEtz is 10KB 2024-03-29T05:13:57.127 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.frf9ahbEtz 2024-03-29T05:13:57.132 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-29T05:13:57.188 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:13:57.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-29T05:13:57.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-29T05:13:57.255 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-29T05:13:57.256 DEBUG:teuthology.orchestra.run.smithi123:> sudo update-grub 2024-03-29T05:13:58.394 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub' 2024-03-29T05:13:58.395 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-29T05:13:58.397 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-03-29T05:13:58.586 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-29T05:13:58.592 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-29T05:13:58.874 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-29T05:13:58.879 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-29T05:13:58.923 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-29T05:13:58.928 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-29T05:13:59.082 INFO:teuthology.orchestra.run.smithi123.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-29T05:13:59.083 INFO:teuthology.orchestra.run.smithi123.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-29T05:13:59.083 INFO:teuthology.orchestra.run.smithi123.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-29T05:13:59.093 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-03-29T05:13:59.095 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-03-29T05:14:29.126 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T05:14:29.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-29T05:14:29.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T05:14:47.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-03-29T05:14:56.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-29T05:14:56.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T05:14:59.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-03-29T05:15:11.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-29T05:15:11.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T05:15:29.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-03-29T05:15:44.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-29T05:15:44.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T05:15:45.417 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-29T05:15:46.674 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-29T05:15:46.674 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-29T05:15:46.675 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-03-29T05:15:46.680 INFO:teuthology.orchestra.run.smithi123.stdout:5.15.0-101-generic 2024-03-29T05:15:46.680 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-29T05:15:46.680 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T05:15:46.680 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-29T05:15:47.681 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T05:15:47.681 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T05:15:47.830 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-03-29T05:15:47.854 DEBUG:teuthology.parallel:result is None 2024-03-29T05:15:47.854 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T05:15:47.861 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T05:15:47.862 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T05:15:47.867 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T05:15:47.874 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T05:15:47.880 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T05:15:47.881 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T05:15:48.084 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T05:15:48.091 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T05:15:48.091 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T05:15:48.115 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:15:48.122 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:15:48.123 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T05:15:48.130 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T05:15:48.130 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T05:15:48.246 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T05:15:48.254 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T05:15:48.254 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T05:15:48.259 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T05:15:48.308 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T05:15:48.355 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:15:48.355 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T05:15:48.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-03-29T05:15:48.489 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T05:15:48.496 INFO:teuthology.task.internal:Starting timer... 2024-03-29T05:15:48.496 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T05:15:48.509 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T05:15:48.517 DEBUG:teuthology.task.selinux:Excluding smithi123: OS 'ubuntu' does not support SELinux 2024-03-29T05:15:48.517 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T05:15:48.517 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-29T05:15:48.518 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T05:15:48.518 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T05:15:48.527 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T05:15:48.532 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T05:15:48.664 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T05:15:48.679 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T05:15:48.681 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T05:22:41.716 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2024-03-29T05:22:41.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-29T05:22:41.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T05:22:41.804 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-03-29T05:22:41.879 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-03-29T05:22:41.880 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T05:22:41.891 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T05:22:41.891 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T05:22:41.892 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T05:22:41.952 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-29T05:22:41.952 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Command line: ntpd -gq 2024-03-29T05:22:41.953 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: ---------------------------------------------------- 2024-03-29T05:22:41.953 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: ntp-4 is maintained by Network Time Foundation, 2024-03-29T05:22:41.953 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-29T05:22:41.953 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: corporation. Support and training for ntp-4 are 2024-03-29T05:22:41.953 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: available at https://www.nwtime.org/support 2024-03-29T05:22:41.953 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: ---------------------------------------------------- 2024-03-29T05:22:41.953 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: proto: precision = 0.045 usec (-24) 2024-03-29T05:22:41.954 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: basedate set to 2022-02-04 2024-03-29T05:22:41.954 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: gps base set to 2022-02-06 (week 2196) 2024-03-29T05:22:41.954 INFO:teuthology.orchestra.run.smithi123.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-29T05:22:41.954 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-29T05:22:41.954 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: restrict ::: KOD does nothing without LIMITED. 2024-03-29T05:22:41.955 INFO:teuthology.orchestra.run.smithi123.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-29T05:22:41.956 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Listen and drop on 0 v6wildcard [::]:123 2024-03-29T05:22:41.956 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-29T05:22:41.956 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Listen normally on 2 lo 127.0.0.1:123 2024-03-29T05:22:41.956 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Listen normally on 3 enp3s0f1 172.21.15.123:123 2024-03-29T05:22:41.956 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Listen normally on 4 lo [::1]:123 2024-03-29T05:22:41.956 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3f1%5]:123 2024-03-29T05:22:41.956 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:41 ntpd[17202]: Listening on routing socket on fd #22 for interface updates 2024-03-29T05:22:42.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:42 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:42.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:42 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:42.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:42 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:42.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:42 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:42.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:42 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:42.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:42 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:43.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:43.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:43.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:43.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:43.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:43.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:43.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:43.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:43 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:44.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:44.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:44.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:44.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:44.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:44.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:44.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:44.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:44 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:45 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:45 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:45 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:45 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:45 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:45.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:45 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:46.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:46.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:46 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:47.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:47 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:47.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:47 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:47.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:47 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:47.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:47 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:47.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:47 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:47.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:47 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:48.953 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: can't open /var/log/ntpstats/rawstats.20240329: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: can't open /var/log/ntpstats/peerstats.20240329: Permission denied 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stdout:29 Mar 05:22:48 ntpd[17202]: ntpd: time slew -0.001058 s 2024-03-29T05:22:48.954 INFO:teuthology.orchestra.run.smithi123.stdout:ntpd: time slew -0.001058s 2024-03-29T05:22:48.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-29T05:22:48.955 INFO:teuthology.orchestra.run.smithi123.stderr:29 Mar 05:22:48 ntpd[17202]: can't open /var/log/ntpstats/loopstats.20240329: Permission denied 2024-03-29T05:22:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2024-03-29T05:22:49.007 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2024-03-29T05:22:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-29T05:22:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-29T05:22:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-29T05:22:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-29T05:22:49.008 INFO:teuthology.run_tasks:Running task install... 2024-03-29T05:22:49.019 DEBUG:teuthology.task.install:project ceph 2024-03-29T05:22:49.019 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8da23624bc17fe0b1f797227e72a0db0ce7821ba'}} 2024-03-29T05:22:49.019 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8da23624bc17fe0b1f797227e72a0db0ce7821ba'} 2024-03-29T05:22:49.019 INFO:teuthology.task.install:Using flavor: default 2024-03-29T05:22:49.025 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T05:22:49.025 INFO:teuthology.task.install:extra packages: [] 2024-03-29T05:22:49.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-key list | grep Ceph 2024-03-29T05:22:49.103 INFO:teuthology.orchestra.run.smithi123.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-29T05:22:49.131 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-29T05:22:49.131 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph.com (release key) 2024-03-29T05:22:49.131 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-29T05:22:49.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8da23624bc17fe0b1f797227e72a0db0ce7821ba 2024-03-29T05:22:49.456 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default/ 2024-03-29T05:22:49.456 INFO:teuthology.task.install.deb:Package version is 19.0.0-1959-g8da23624-1jammy 2024-03-29T05:22:49.566 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:22:49.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-29T05:22:49.580 DEBUG:teuthology.orchestra.run.smithi123:> sudo apt-get update 2024-03-29T05:22:49.756 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-29T05:22:49.770 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-29T05:22:49.792 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-29T05:22:49.865 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy InRelease 2024-03-29T05:22:49.896 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-29T05:22:49.914 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-29T05:22:49.961 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-29T05:22:50.003 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-29T05:22:50.047 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-03-29T05:22:50.117 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 28.9 kB in 0s (79.6 kB/s) 2024-03-29T05:22:51.051 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-03-29T05:22:51.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1959-g8da23624-1jammy cephadm=19.0.0-1959-g8da23624-1jammy ceph-mds=19.0.0-1959-g8da23624-1jammy ceph-mgr=19.0.0-1959-g8da23624-1jammy ceph-common=19.0.0-1959-g8da23624-1jammy ceph-fuse=19.0.0-1959-g8da23624-1jammy ceph-test=19.0.0-1959-g8da23624-1jammy ceph-volume=19.0.0-1959-g8da23624-1jammy radosgw=19.0.0-1959-g8da23624-1jammy python3-rados=19.0.0-1959-g8da23624-1jammy python3-rgw=19.0.0-1959-g8da23624-1jammy python3-cephfs=19.0.0-1959-g8da23624-1jammy python3-rbd=19.0.0-1959-g8da23624-1jammy libcephfs2=19.0.0-1959-g8da23624-1jammy libcephfs-dev=19.0.0-1959-g8da23624-1jammy librados2=19.0.0-1959-g8da23624-1jammy librbd1=19.0.0-1959-g8da23624-1jammy rbd-fuse=19.0.0-1959-g8da23624-1jammy 2024-03-29T05:22:51.133 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2024-03-29T05:22:51.330 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2024-03-29T05:22:51.331 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2024-03-29T05:22:51.513 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages were automatically installed and are no longer required: 2024-03-29T05:22:51.513 INFO:teuthology.orchestra.run.smithi123.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-29T05:22:51.513 INFO:teuthology.orchestra.run.smithi123.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-29T05:22:51.513 INFO:teuthology.orchestra.run.smithi123.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-29T05:22:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-29T05:22:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-29T05:22:51.514 INFO:teuthology.orchestra.run.smithi123.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-29T05:22:51.514 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2024-03-29T05:22:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-29T05:22:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-29T05:22:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-29T05:22:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-29T05:22:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-29T05:22:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-29T05:22:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-29T05:22:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-29T05:22:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet zip 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-29T05:22:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-29T05:22:51.519 INFO:teuthology.orchestra.run.smithi123.stdout:Recommended packages: 2024-03-29T05:22:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: btrfs-tools 2024-03-29T05:22:51.623 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2024-03-29T05:22:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-29T05:22:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-29T05:22:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-29T05:22:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-29T05:22:51.761 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 118 newly installed, 0 to remove and 298 not upgraded. 2024-03-29T05:22:51.761 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 178 MB of archives. 2024-03-29T05:22:51.761 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 815 MB of additional disk space will be used. 2024-03-29T05:22:51.762 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1959-g8da23624-1jammy [3,544 kB] 2024-03-29T05:22:51.808 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-29T05:22:52.014 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1959-g8da23624-1jammy [3,217 kB] 2024-03-29T05:22:52.126 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1959-g8da23624-1jammy [711 kB] 2024-03-29T05:22:52.147 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1959-g8da23624-1jammy [356 kB] 2024-03-29T05:22:52.161 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1959-g8da23624-1jammy [32.8 kB] 2024-03-29T05:22:52.161 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1959-g8da23624-1jammy [180 kB] 2024-03-29T05:22:52.165 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1959-g8da23624-1jammy [64.6 kB] 2024-03-29T05:22:52.167 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1959-g8da23624-1jammy [330 kB] 2024-03-29T05:22:52.176 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1959-g8da23624-1jammy [6,577 kB] 2024-03-29T05:22:52.235 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-29T05:22:52.247 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-29T05:22:52.329 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-29T05:22:52.393 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1959-g8da23624-1jammy [112 kB] 2024-03-29T05:22:52.396 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1959-g8da23624-1jammy [466 kB] 2024-03-29T05:22:52.411 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1959-g8da23624-1jammy [25.2 MB] 2024-03-29T05:22:52.558 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-29T05:22:52.570 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-29T05:22:52.601 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-29T05:22:52.610 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-29T05:22:52.611 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-29T05:22:52.612 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-29T05:22:52.631 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-29T05:22:52.799 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-29T05:22:53.206 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-29T05:22:53.240 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-29T05:22:53.311 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-29T05:22:53.379 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1959-g8da23624-1jammy [5,077 kB] 2024-03-29T05:22:53.379 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-29T05:22:53.385 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-29T05:22:53.407 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-29T05:22:53.414 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-29T05:22:53.415 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-29T05:22:53.426 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-29T05:22:53.427 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-29T05:22:53.457 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1959-g8da23624-1jammy [244 kB] 2024-03-29T05:22:53.459 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1959-g8da23624-1jammy [124 kB] 2024-03-29T05:22:53.461 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1959-g8da23624-1jammy [1,526 kB] 2024-03-29T05:22:53.491 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1959-g8da23624-1jammy [6,247 kB] 2024-03-29T05:22:53.593 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-29T05:22:53.676 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-29T05:22:53.704 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-29T05:22:53.705 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1959-g8da23624-1jammy [22.7 MB] 2024-03-29T05:22:53.721 INFO:teuthology.orchestra.run.smithi123.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-29T05:22:53.766 INFO:teuthology.orchestra.run.smithi123.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-29T05:22:53.981 INFO:teuthology.orchestra.run.smithi123.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-29T05:22:53.996 INFO:teuthology.orchestra.run.smithi123.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-29T05:22:54.002 INFO:teuthology.orchestra.run.smithi123.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-29T05:22:54.008 INFO:teuthology.orchestra.run.smithi123.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-29T05:22:54.029 INFO:teuthology.orchestra.run.smithi123.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-29T05:22:54.033 INFO:teuthology.orchestra.run.smithi123.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-29T05:22:54.199 INFO:teuthology.orchestra.run.smithi123.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-29T05:22:54.531 INFO:teuthology.orchestra.run.smithi123.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-29T05:22:54.599 INFO:teuthology.orchestra.run.smithi123.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-29T05:22:54.600 INFO:teuthology.orchestra.run.smithi123.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-29T05:22:54.601 INFO:teuthology.orchestra.run.smithi123.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1959-g8da23624-1jammy [14.0 kB] 2024-03-29T05:22:54.601 INFO:teuthology.orchestra.run.smithi123.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1959-g8da23624-1jammy [869 kB] 2024-03-29T05:22:54.618 INFO:teuthology.orchestra.run.smithi123.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1959-g8da23624-1jammy [2,434 kB] 2024-03-29T05:22:54.661 INFO:teuthology.orchestra.run.smithi123.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1959-g8da23624-1jammy [777 kB] 2024-03-29T05:22:54.691 INFO:teuthology.orchestra.run.smithi123.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1959-g8da23624-1jammy [145 kB] 2024-03-29T05:22:54.699 INFO:teuthology.orchestra.run.smithi123.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1959-g8da23624-1jammy [2,226 kB] 2024-03-29T05:22:54.778 INFO:teuthology.orchestra.run.smithi123.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1959-g8da23624-1jammy [8,625 kB] 2024-03-29T05:22:54.780 INFO:teuthology.orchestra.run.smithi123.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-29T05:22:54.781 INFO:teuthology.orchestra.run.smithi123.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-29T05:22:54.787 INFO:teuthology.orchestra.run.smithi123.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-29T05:22:54.793 INFO:teuthology.orchestra.run.smithi123.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-29T05:22:54.809 INFO:teuthology.orchestra.run.smithi123.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-29T05:22:54.831 INFO:teuthology.orchestra.run.smithi123.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-29T05:22:54.837 INFO:teuthology.orchestra.run.smithi123.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-29T05:22:55.013 INFO:teuthology.orchestra.run.smithi123.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-29T05:22:55.151 INFO:teuthology.orchestra.run.smithi123.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1959-g8da23624-1jammy [14.2 kB] 2024-03-29T05:22:55.152 INFO:teuthology.orchestra.run.smithi123.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1959-g8da23624-1jammy [50.0 MB] 2024-03-29T05:22:55.270 INFO:teuthology.orchestra.run.smithi123.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-29T05:22:55.449 INFO:teuthology.orchestra.run.smithi123.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-29T05:22:55.462 INFO:teuthology.orchestra.run.smithi123.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-29T05:22:55.483 INFO:teuthology.orchestra.run.smithi123.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-29T05:22:55.547 INFO:teuthology.orchestra.run.smithi123.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-29T05:22:55.836 INFO:teuthology.orchestra.run.smithi123.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-29T05:22:55.838 INFO:teuthology.orchestra.run.smithi123.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-29T05:22:55.841 INFO:teuthology.orchestra.run.smithi123.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-29T05:22:55.841 INFO:teuthology.orchestra.run.smithi123.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-29T05:22:55.844 INFO:teuthology.orchestra.run.smithi123.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-29T05:22:56.064 INFO:teuthology.orchestra.run.smithi123.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-29T05:22:56.497 INFO:teuthology.orchestra.run.smithi123.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-29T05:22:56.503 INFO:teuthology.orchestra.run.smithi123.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-29T05:22:56.792 INFO:teuthology.orchestra.run.smithi123.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-29T05:22:56.793 INFO:teuthology.orchestra.run.smithi123.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-29T05:22:56.795 INFO:teuthology.orchestra.run.smithi123.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-29T05:22:56.796 INFO:teuthology.orchestra.run.smithi123.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-29T05:22:56.798 INFO:teuthology.orchestra.run.smithi123.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-29T05:22:56.800 INFO:teuthology.orchestra.run.smithi123.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-29T05:22:56.801 INFO:teuthology.orchestra.run.smithi123.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-29T05:22:56.803 INFO:teuthology.orchestra.run.smithi123.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-29T05:22:56.984 INFO:teuthology.orchestra.run.smithi123.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-29T05:22:57.400 INFO:teuthology.orchestra.run.smithi123.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-29T05:22:57.476 INFO:teuthology.orchestra.run.smithi123.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-29T05:22:57.545 INFO:teuthology.orchestra.run.smithi123.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-29T05:22:57.633 INFO:teuthology.orchestra.run.smithi123.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-29T05:22:57.633 INFO:teuthology.orchestra.run.smithi123.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1959-g8da23624-1jammy [121 kB] 2024-03-29T05:22:57.634 INFO:teuthology.orchestra.run.smithi123.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1959-g8da23624-1jammy [38.9 kB] 2024-03-29T05:22:57.635 INFO:teuthology.orchestra.run.smithi123.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1959-g8da23624-1jammy [13.0 MB] 2024-03-29T05:22:57.644 INFO:teuthology.orchestra.run.smithi123.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-29T05:22:57.654 INFO:teuthology.orchestra.run.smithi123.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-29T05:22:57.658 INFO:teuthology.orchestra.run.smithi123.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-29T05:22:57.708 INFO:teuthology.orchestra.run.smithi123.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-29T05:22:57.712 INFO:teuthology.orchestra.run.smithi123.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-29T05:22:57.737 INFO:teuthology.orchestra.run.smithi123.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-29T05:22:57.901 INFO:teuthology.orchestra.run.smithi123.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-29T05:22:58.123 INFO:teuthology.orchestra.run.smithi123.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/squid/8da23624bc17fe0b1f797227e72a0db0ce7821ba/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1959-g8da23624-1jammy [91.4 kB] 2024-03-29T05:22:58.205 INFO:teuthology.orchestra.run.smithi123.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-29T05:22:58.223 INFO:teuthology.orchestra.run.smithi123.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-29T05:22:58.376 INFO:teuthology.orchestra.run.smithi123.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-29T05:22:58.406 INFO:teuthology.orchestra.run.smithi123.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-29T05:22:58.563 INFO:teuthology.orchestra.run.smithi123.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-29T05:22:58.565 INFO:teuthology.orchestra.run.smithi123.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-29T05:22:58.567 INFO:teuthology.orchestra.run.smithi123.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-29T05:22:58.590 INFO:teuthology.orchestra.run.smithi123.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-29T05:22:58.594 INFO:teuthology.orchestra.run.smithi123.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-29T05:22:59.266 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 178 MB in 7s (25.1 MB/s) 2024-03-29T05:22:59.430 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-29T05:22:59.531 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-29T05:22:59.536 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-29T05:22:59.645 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-29T05:22:59.965 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-29T05:22:59.992 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-29T05:23:00.031 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-29T05:23:00.310 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-29T05:23:00.336 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-29T05:23:00.376 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-29T05:23:00.813 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-29T05:23:00.840 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-29T05:23:00.888 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-29T05:23:01.299 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-29T05:23:01.325 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-29T05:23:01.365 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-29T05:23:01.669 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-29T05:23:01.698 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-29T05:23:01.727 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-29T05:23:02.072 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-29T05:23:02.098 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-29T05:23:02.138 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-29T05:23:02.449 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librados2. 2024-03-29T05:23:02.475 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../007-librados2_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:02.516 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librados2 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:02.935 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librbd1. 2024-03-29T05:23:02.950 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../008-librbd1_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:02.985 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librbd1 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:03.379 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs2. 2024-03-29T05:23:03.406 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:03.446 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs2 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:03.723 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rados. 2024-03-29T05:23:03.739 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:03.773 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rados (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:04.043 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-29T05:23:04.058 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:04.093 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-ceph-argparse (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:04.337 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cephfs. 2024-03-29T05:23:04.363 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:04.403 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cephfs (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:04.714 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-29T05:23:04.741 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:04.780 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-ceph-common (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:05.077 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-29T05:23:05.104 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-29T05:23:05.136 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-29T05:23:05.597 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-prettytable. 2024-03-29T05:23:05.623 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-29T05:23:05.655 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-29T05:23:05.907 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rbd. 2024-03-29T05:23:05.922 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:05.965 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rbd (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:06.310 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-29T05:23:06.336 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-29T05:23:06.385 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-29T05:23:06.712 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-29T05:23:06.739 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-29T05:23:06.796 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-29T05:23:07.090 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-29T05:23:07.116 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-29T05:23:07.148 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-29T05:23:07.468 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua5.1. 2024-03-29T05:23:07.494 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-29T05:23:07.534 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-29T05:23:07.846 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua-any. 2024-03-29T05:23:07.872 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-29T05:23:07.913 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-29T05:23:08.156 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package zip. 2024-03-29T05:23:08.171 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-29T05:23:08.215 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking zip (3.0-12build2) ... 2024-03-29T05:23:08.560 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package unzip. 2024-03-29T05:23:08.586 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-29T05:23:08.617 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-29T05:23:09.038 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package luarocks. 2024-03-29T05:23:09.064 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-29T05:23:09.105 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-29T05:23:09.487 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librgw2. 2024-03-29T05:23:09.514 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../025-librgw2_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:09.554 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librgw2 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:10.064 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rgw. 2024-03-29T05:23:10.091 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:10.131 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rgw (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:10.467 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-29T05:23:10.493 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-29T05:23:10.534 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-29T05:23:10.803 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libradosstriper1. 2024-03-29T05:23:10.819 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:10.853 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libradosstriper1 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:11.273 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-common. 2024-03-29T05:23:11.300 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:11.340 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-common (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:12.678 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-base. 2024-03-29T05:23:12.705 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:12.753 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-base (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:13.274 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-29T05:23:13.300 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-29T05:23:13.340 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-29T05:23:13.643 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cheroot. 2024-03-29T05:23:13.670 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-29T05:23:13.710 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-29T05:23:14.021 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-29T05:23:14.047 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-29T05:23:14.088 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-29T05:23:14.340 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-29T05:23:14.367 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-29T05:23:14.407 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-29T05:23:14.693 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-29T05:23:14.719 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-29T05:23:14.759 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-29T05:23:15.062 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-tempora. 2024-03-29T05:23:15.088 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-29T05:23:15.129 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-29T05:23:15.398 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-portend. 2024-03-29T05:23:15.425 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-29T05:23:15.456 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-29T05:23:15.709 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-29T05:23:15.724 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-29T05:23:15.967 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-29T05:23:16.262 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-29T05:23:16.289 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-29T05:23:16.328 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-29T05:23:16.690 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-natsort. 2024-03-29T05:23:16.717 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-29T05:23:16.757 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-29T05:23:17.051 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-logutils. 2024-03-29T05:23:17.078 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-29T05:23:17.118 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-29T05:23:17.379 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-29T05:23:17.394 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-29T05:23:17.429 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-29T05:23:17.748 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-mako. 2024-03-29T05:23:17.775 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-29T05:23:17.815 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-29T05:23:18.143 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-29T05:23:18.170 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-29T05:23:18.209 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-29T05:23:18.512 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-29T05:23:18.539 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-29T05:23:18.579 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-29T05:23:18.848 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-webob. 2024-03-29T05:23:18.875 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-29T05:23:18.906 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-29T05:23:19.193 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-waitress. 2024-03-29T05:23:19.219 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-29T05:23:19.254 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-29T05:23:19.537 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-tempita. 2024-03-29T05:23:19.564 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-29T05:23:19.604 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-29T05:23:19.915 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-paste. 2024-03-29T05:23:19.942 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-29T05:23:19.981 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-29T05:23:20.293 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-29T05:23:20.308 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-29T05:23:20.350 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-29T05:23:20.637 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-29T05:23:20.664 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-29T05:23:20.703 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-29T05:23:21.006 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-webtest. 2024-03-29T05:23:21.033 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-29T05:23:21.073 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-29T05:23:21.384 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pecan. 2024-03-29T05:23:21.411 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-29T05:23:21.450 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-29T05:23:21.780 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-certifi. 2024-03-29T05:23:21.796 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-29T05:23:21.830 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-29T05:23:22.116 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-idna. 2024-03-29T05:23:22.143 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-29T05:23:22.183 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-29T05:23:22.485 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-urllib3. 2024-03-29T05:23:22.501 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-29T05:23:22.543 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-29T05:23:22.864 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-requests. 2024-03-29T05:23:22.891 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-29T05:23:22.947 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-29T05:23:23.258 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-29T05:23:23.285 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-29T05:23:23.324 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-29T05:23:23.610 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-29T05:23:23.638 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:23.677 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:24.057 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-29T05:23:24.084 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:24.123 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:24.434 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-29T05:23:24.461 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-29T05:23:24.501 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-29T05:23:24.804 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-openssl. 2024-03-29T05:23:24.819 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-29T05:23:24.853 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-29T05:23:25.941 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr. 2024-03-29T05:23:25.967 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:26.007 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:26.602 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mon. 2024-03-29T05:23:26.629 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:26.668 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mon (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:27.213 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-osd. 2024-03-29T05:23:27.240 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:27.271 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-osd (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:28.366 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph. 2024-03-29T05:23:28.381 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../066-ceph_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:28.416 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:28.677 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-fuse. 2024-03-29T05:23:28.703 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:28.735 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-fuse (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:29.138 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mds. 2024-03-29T05:23:29.165 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:29.205 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mds (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:29.608 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package cephadm. 2024-03-29T05:23:29.634 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../069-cephadm_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:29.674 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking cephadm (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:29.943 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-29T05:23:29.970 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-29T05:23:30.010 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-29T05:23:30.337 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-29T05:23:30.353 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-29T05:23:30.396 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-29T05:23:31.434 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-babel. 2024-03-29T05:23:31.461 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-29T05:23:31.500 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-29T05:23:31.828 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jinja2. 2024-03-29T05:23:31.855 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-29T05:23:31.911 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-29T05:23:32.273 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-29T05:23:32.299 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:32.339 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:32.676 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-29T05:23:32.702 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-29T05:23:32.742 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-29T05:23:33.053 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-routes. 2024-03-29T05:23:33.080 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-29T05:23:33.120 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-29T05:23:33.456 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-29T05:23:33.483 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:33.522 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:34.686 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-29T05:23:34.713 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-29T05:23:34.752 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-29T05:23:35.231 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-joblib. 2024-03-29T05:23:35.258 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-29T05:23:35.297 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-29T05:23:35.626 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-29T05:23:35.644 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-29T05:23:35.675 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-29T05:23:35.970 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-sklearn. 2024-03-29T05:23:35.998 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-29T05:23:36.036 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-29T05:23:36.761 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-29T05:23:36.789 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:36.905 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:37.900 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cachetools. 2024-03-29T05:23:37.927 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-29T05:23:37.966 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-29T05:23:38.219 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-29T05:23:38.243 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-29T05:23:38.277 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-29T05:23:38.581 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-29T05:23:38.608 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-29T05:23:38.647 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-29T05:23:38.958 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rsa. 2024-03-29T05:23:38.985 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-29T05:23:39.042 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-29T05:23:39.370 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-google-auth. 2024-03-29T05:23:39.397 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-29T05:23:39.437 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-29T05:23:39.757 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-29T05:23:39.784 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-29T05:23:39.823 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-29T05:23:40.101 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-websocket. 2024-03-29T05:23:40.128 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-29T05:23:40.159 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-29T05:23:40.421 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-29T05:23:40.448 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-29T05:23:40.487 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-29T05:23:41.168 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-29T05:23:41.196 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:41.234 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:41.512 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-29T05:23:41.540 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-29T05:23:41.570 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-29T05:23:42.065 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-29T05:23:42.092 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-29T05:23:42.131 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-29T05:23:42.384 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package jq. 2024-03-29T05:23:42.412 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-29T05:23:42.450 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-29T05:23:42.719 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package socat. 2024-03-29T05:23:42.743 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-29T05:23:42.786 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-29T05:23:43.056 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package xmlstarlet. 2024-03-29T05:23:43.072 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-29T05:23:43.105 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-29T05:23:43.426 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-test. 2024-03-29T05:23:43.442 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:43.484 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-test (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:45.624 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-volume. 2024-03-29T05:23:45.640 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1959-g8da23624-1jammy_all.deb ... 2024-03-29T05:23:45.674 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-volume (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:45.944 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-29T05:23:45.960 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:45.994 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs-dev (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:46.288 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-29T05:23:46.316 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-29T05:23:46.355 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-29T05:23:46.851 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-29T05:23:46.878 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-29T05:23:46.917 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-29T05:23:47.505 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package nvme-cli. 2024-03-29T05:23:47.533 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-29T05:23:47.571 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-29T05:23:47.926 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package pkg-config. 2024-03-29T05:23:47.954 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-29T05:23:47.993 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-29T05:23:48.254 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-29T05:23:48.270 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-29T05:23:48.312 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-29T05:23:48.716 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-29T05:23:48.744 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-29T05:23:48.782 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-29T05:23:49.085 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pastescript. 2024-03-29T05:23:49.112 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-29T05:23:49.151 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-29T05:23:49.471 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pluggy. 2024-03-29T05:23:49.499 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-29T05:23:49.538 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-29T05:23:49.874 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-psutil. 2024-03-29T05:23:49.902 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-29T05:23:49.941 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-29T05:23:50.294 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-py. 2024-03-29T05:23:50.322 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-29T05:23:50.368 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-29T05:23:50.688 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pygments. 2024-03-29T05:23:50.716 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-29T05:23:50.755 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-29T05:23:51.209 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-29T05:23:51.237 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-29T05:23:51.275 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-29T05:23:51.545 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-toml. 2024-03-29T05:23:51.561 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-29T05:23:51.594 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-29T05:23:51.848 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pytest. 2024-03-29T05:23:51.875 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-29T05:23:51.914 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-29T05:23:52.268 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-simplejson. 2024-03-29T05:23:52.296 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-29T05:23:52.334 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-29T05:23:52.586 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-29T05:23:52.602 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-29T05:23:52.636 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-29T05:23:53.349 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package radosgw. 2024-03-29T05:23:53.377 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../116-radosgw_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:53.423 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking radosgw (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:54.201 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package rbd-fuse. 2024-03-29T05:23:54.218 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1959-g8da23624-1jammy_amd64.deb ... 2024-03-29T05:23:54.259 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking rbd-fuse (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:54.572 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-29T05:23:54.848 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-29T05:23:54.974 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-29T05:23:55.231 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-29T05:23:55.567 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-29T05:23:56.020 INFO:teuthology.orchestra.run.smithi123.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-29T05:23:56.076 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up cephadm (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:23:56.534 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user cephadm....done 2024-03-29T05:23:56.707 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-29T05:23:57.000 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-29T05:23:57.226 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-29T05:23:57.343 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-29T05:23:57.595 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-29T05:23:57.855 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-29T05:23:57.989 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-29T05:23:58.290 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-29T05:23:58.650 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-29T05:23:58.894 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-29T05:23:59.161 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-29T05:23:59.421 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-29T05:23:59.672 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-29T05:23:59.932 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-29T05:24:00.183 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-ceph-argparse (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:00.425 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-29T05:24:00.526 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-29T05:24:00.660 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-29T05:24:00.786 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-29T05:24:01.046 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-29T05:24:01.437 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-29T05:24:01.468 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-29T05:24:01.532 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-29T05:24:01.650 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-29T05:24:02.001 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-29T05:24:02.110 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-29T05:24:02.378 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-29T05:24:02.646 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-29T05:24:03.264 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up zip (3.0-12build2) ... 2024-03-29T05:24:03.390 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-29T05:24:04.075 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-29T05:24:04.342 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-29T05:24:04.469 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-29T05:24:04.594 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-29T05:24:04.887 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-29T05:24:05.272 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-29T05:24:05.514 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-29T05:24:05.882 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-29T05:24:06.142 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-29T05:24:06.469 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-29T05:24:06.720 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-29T05:24:06.846 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-ceph-common (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:07.139 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-idna (3.3-1) ... 2024-03-29T05:24:07.456 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-29T05:24:08.654 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-29T05:24:08.913 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-29T05:24:09.048 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-29T05:24:09.324 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-29T05:24:09.450 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-29T05:24:09.551 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-29T05:24:09.835 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-29T05:24:10.103 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-29T05:24:10.362 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-29T05:24:10.497 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-29T05:24:10.798 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-29T05:24:11.049 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-29T05:24:11.309 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-29T05:24:11.560 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-29T05:24:11.811 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-29T05:24:12.070 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-29T05:24:12.196 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-29T05:24:12.481 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-29T05:24:12.598 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-29T05:24:12.883 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-29T05:24:13.159 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-29T05:24:13.385 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-29T05:24:13.486 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-29T05:24:13.612 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-29T05:24:13.700 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-29T05:24:13.922 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-29T05:24:14.048 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-29T05:24:14.441 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-29T05:24:14.685 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-29T05:24:14.801 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-29T05:24:15.086 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-29T05:24:15.212 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-29T05:24:15.546 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-29T05:24:15.856 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-29T05:24:16.140 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-29T05:24:16.266 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-29T05:24:16.551 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-29T05:24:16.802 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-29T05:24:17.138 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-29T05:24:17.515 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-29T05:24:17.791 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-29T05:24:18.110 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-29T05:24:18.228 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librados2 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:18.354 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:18.505 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-29T05:24:18.791 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs2 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:18.909 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libradosstriper1 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:19.035 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-29T05:24:19.303 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librbd1 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:19.429 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-modules-core (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:19.564 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-fuse (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:19.773 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-29T05:24:19.773 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-29T05:24:20.240 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs-dev (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:20.341 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rados (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:20.450 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librgw2 (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:20.576 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-29T05:24:21.821 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rbd (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:21.956 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up rbd-fuse (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:22.107 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rgw (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:22.207 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cephfs (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:22.317 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-common (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:22.834 INFO:teuthology.orchestra.run.smithi123.stdout:Adding group ceph....done 2024-03-29T05:24:23.211 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user ceph....done 2024-03-29T05:24:23.346 INFO:teuthology.orchestra.run.smithi123.stdout:Setting system user ceph properties....done 2024-03-29T05:24:23.361 INFO:teuthology.orchestra.run.smithi123.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-29T05:24:23.495 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-29T05:24:23.839 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-29T05:24:24.301 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-test (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:24.427 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up radosgw (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:24.882 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-29T05:24:24.882 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-29T05:24:25.404 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-base (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:25.765 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-29T05:24:26.253 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mds (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:26.562 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-29T05:24:26.562 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-29T05:24:27.042 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:27.241 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-29T05:24:27.241 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-29T05:24:27.735 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-osd (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:28.012 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-29T05:24:28.012 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-29T05:24:28.511 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:28.629 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:28.781 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mon (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:28.993 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-29T05:24:28.994 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-29T05:24:29.499 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-cephadm (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:29.642 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:29.768 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-dashboard (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:29.894 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-volume (19.0.0-1959-g8da23624-1jammy) ... 2024-03-29T05:24:30.222 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-29T05:24:30.322 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-29T05:24:30.458 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-29T05:24:31.226 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-29T05:24:31.862 INFO:teuthology.orchestra.run.smithi123.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-29T05:24:31.867 DEBUG:teuthology.parallel:result is None 2024-03-29T05:24:31.868 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8da23624bc17fe0b1f797227e72a0db0ce7821ba 2024-03-29T05:24:32.004 DEBUG:teuthology.orchestra.run.smithi123:> dpkg-query -W -f '${Version}' ceph 2024-03-29T05:24:32.022 INFO:teuthology.orchestra.run.smithi123.stdout:19.0.0-1959-g8da23624-1jammy 2024-03-29T05:24:32.023 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1959-g8da23624-1jammy 2024-03-29T05:24:32.023 INFO:teuthology.task.install:The correct ceph version 19.0.0-1959-g8da23624-1jammy is installed. 2024-03-29T05:24:32.025 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T05:24:32.025 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:24:32.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T05:24:32.077 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T05:24:32.077 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:24:32.077 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T05:24:32.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T05:24:32.188 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T05:24:32.188 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:24:32.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T05:24:32.241 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T05:24:32.297 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T05:24:32.298 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:24:32.298 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T05:24:32.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T05:24:32.409 INFO:teuthology.run_tasks:Running task workunit... 2024-03-29T05:24:32.426 INFO:tasks.workunit:Pulling workunits from ref 8da23624bc17fe0b1f797227e72a0db0ce7821ba 2024-03-29T05:24:32.426 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-29T05:24:32.426 INFO:tasks.workunit:timeout=3h 2024-03-29T05:24:32.426 INFO:tasks.workunit:cleanup=True 2024-03-29T05:24:32.426 DEBUG:teuthology.orchestra.run.smithi123:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-29T05:24:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T05:24:32.458 INFO:teuthology.orchestra.run.smithi123.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-29T05:24:32.459 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-29T05:24:32.506 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-29T05:24:32.506 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-29T05:24:32.554 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8da23624bc17fe0b1f797227e72a0db0ce7821ba 2024-03-29T05:24:32.604 INFO:tasks.workunit.client.0.smithi123.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-29T05:25:36.624 INFO:tasks.workunit.client.0.smithi123.stderr:Note: switching to '8da23624bc17fe0b1f797227e72a0db0ce7821ba'. 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr:state without impacting any branches by switching back to a branch. 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr: git switch -c 2024-03-29T05:25:36.625 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-29T05:25:36.626 INFO:tasks.workunit.client.0.smithi123.stderr:Or undo this operation with: 2024-03-29T05:25:36.626 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-29T05:25:36.626 INFO:tasks.workunit.client.0.smithi123.stderr: git switch - 2024-03-29T05:25:36.626 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-29T05:25:36.626 INFO:tasks.workunit.client.0.smithi123.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-29T05:25:36.626 INFO:tasks.workunit.client.0.smithi123.stderr: 2024-03-29T05:25:36.626 INFO:tasks.workunit.client.0.smithi123.stderr:HEAD is now at 8da23624bc1 Merge pull request #56502 from rhcs-dashboard/wip-65166-squid 2024-03-29T05:25:36.628 DEBUG:teuthology.orchestra.run.smithi123:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-29T05:25:36.674 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-03-29T05:25:36.674 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-29T05:25:36.721 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-03-29T05:25:36.721 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-03-29T05:25:36.722 DEBUG:teuthology.orchestra.run.smithi123:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-03-29T05:25:36.794 INFO:tasks.workunit.client.0.smithi123.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-03-29T05:25:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-03-29T05:25:36.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-03-29T05:25:36.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-03-29T05:25:36.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-03-29T05:25:36.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none ' 2024-03-29T05:25:36.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-03-29T05:25:36.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-29T05:25:36.802 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-03-29T05:25:36.802 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-29T05:25:36.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:25:36.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:25:36.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:25:36.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:25:36.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:25:36.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:25:36.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:25:36.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:25:36.808 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:25:36.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:25:36.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:25:36.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:25:36.809 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:25:36.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:25:36.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:25:36.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-29T05:25:36.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:25:36.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:25:36.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:25:36.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20229 2024-03-29T05:25:36.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:25:36.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:25:36.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-03-29T05:25:36.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:25:36.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:25:36.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:25:36.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20229 2024-03-29T05:25:36.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:25:36.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-29T05:25:36.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-29T05:25:36.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:25:36.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-03-29T05:25:36.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-03-29T05:25:36.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-03-29T05:25:36.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-03-29T05:25:36.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-03-29T05:25:36.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-03-29T05:25:36.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-03-29T05:25:36.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-03-29T05:25:36.817 INFO:tasks.workunit.client.0.smithi123.stdout:Dir: td/osd-mapper 2024-03-29T05:25:36.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-29T05:25:36.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-29T05:25:38.695 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 c20f610e-1e90-403d-a701-4f1b3b6a04bf 2024-03-29T05:25:39.020 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:25:39.099 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:39.095+0000 7fd389abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:25:39.101 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:39.099+0000 7fd389abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:25:39.103 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:39.099+0000 7fd389abf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:25:39.104 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:39.099+0000 7fd389abf7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-29T05:25:39.104 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:39.099+0000 7fd389abf7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-29T05:25:39.104 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:39.099+0000 7fd389abf7c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-29T05:25:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:39.187+0000 7fd389abf7c0 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-03-29T05:25:46.002 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:25:46.191 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:25:46.218 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:46.215+0000 7eff1afbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:25:46.221 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:46.219+0000 7eff1afbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:25:46.222 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:46.219+0000 7eff1afbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:25:46.404 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:25:46.779 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:46.775+0000 7eff1afbe7c0 -1 Falling back to public interface 2024-03-29T05:25:47.611 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:25:48.811 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:25:49.526 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:25:49.523+0000 7eff1afbe7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:25:50.125 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:25:51.541 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:25:52.860 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:25:54.179 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:25:55.539 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:25:56.843 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:25:58.159 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:25:59.532 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:26:00.834 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:26:02.149 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:26:03.470 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:26:05.247 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:26:07.032 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:26:09.063 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:26:10.807 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:26:12.440 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:26:13.837 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:26:15.629 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:26:17.470 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T05:26:18.990 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T05:26:19.300 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/473892094,v1:127.0.0.1:6801/473892094] [v2:127.0.0.1:6802/473892094,v1:127.0.0.1:6803/473892094] exists,up c20f610e-1e90-403d-a701-4f1b3b6a04bf 2024-03-29T05:26:19.305 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 74e41364-61f8-436f-a72f-9a43b5ce2757 2024-03-29T05:26:19.723 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:26:19.781 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:19.777+0000 7fea529207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:19.826 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:19.825+0000 7fea529207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:19.827 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:19.825+0000 7fea529207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:19.828 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:19.825+0000 7fea529207c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-29T05:26:19.828 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:19.825+0000 7fea529207c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-29T05:26:19.828 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:19.825+0000 7fea529207c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-29T05:26:19.905 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:19.901+0000 7fea529207c0 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-03-29T05:26:26.688 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:26:27.076 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:26:27.111 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:27.109+0000 7ff6671e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:27.141 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:27.137+0000 7ff6671e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:27.142 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:27.141+0000 7ff6671e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:27.420 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:26:27.720 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:27.716+0000 7ff6671e37c0 -1 Falling back to public interface 2024-03-29T05:26:28.759 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:26:30.077 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:26:30.419 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:30.416+0000 7ff6671e37c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:26:31.392 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:26:32.773 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:26:34.086 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:26:35.517 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:26:36.879 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:26:38.295 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:26:39.620 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:26:40.938 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:26:42.247 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:26:43.685 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:26:44.998 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:26:46.574 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:26:48.478 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:26:50.130 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:26:51.664 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:26:53.179 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:26:54.907 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:26:56.683 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:26:56.997 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1700515174,v1:127.0.0.1:6811/1700515174] [v2:127.0.0.1:6812/1700515174,v1:127.0.0.1:6813/1700515174] exists,up 74e41364-61f8-436f-a72f-9a43b5ce2757 2024-03-29T05:26:57.003 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 bd922523-731c-4954-89b2-d8227a45a7d3 2024-03-29T05:26:57.414 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:26:57.477 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:57.467+0000 7f0001fc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:57.507 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:57.503+0000 7f0001fc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:57.508 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:57.503+0000 7f0001fc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:26:57.509 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:57.503+0000 7f0001fc37c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-29T05:26:57.509 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:57.503+0000 7f0001fc37c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-29T05:26:57.509 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:57.503+0000 7f0001fc37c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-29T05:26:57.583 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:26:57.579+0000 7f0001fc37c0 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-03-29T05:27:04.797 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:27:05.173 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:27:05.208 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:05.203+0000 7f37ae26f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:27:05.239 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:05.235+0000 7f37ae26f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:27:05.240 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:05.235+0000 7f37ae26f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:27:05.504 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:27:05.813 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:05.807+0000 7f37ae26f7c0 -1 Falling back to public interface 2024-03-29T05:27:06.816 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:27:08.131 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:27:08.482 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:08.475+0000 7f37ae26f7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:27:09.449 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:27:10.937 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:27:12.329 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:27:13.836 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:27:15.156 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:27:16.555 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:27:17.873 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:27:19.190 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:27:20.558 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:27:21.873 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:27:23.185 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:27:25.095 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:27:26.695 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:27:28.115 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:27:30.009 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:27:31.882 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:27:33.201 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:27:34.992 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:27:35.315 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3795027281,v1:127.0.0.1:6819/3795027281] [v2:127.0.0.1:6820/3795027281,v1:127.0.0.1:6821/3795027281] exists,up bd922523-731c-4954-89b2-d8227a45a7d3 2024-03-29T05:27:35.956 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'test' created 2024-03-29T05:27:38.287 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836502 2024-03-29T05:27:41.199 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672979 2024-03-29T05:27:41.530 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509447 2024-03-29T05:27:44.179 INFO:tasks.workunit.client.0.smithi123.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-03-29T05:27:44.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-03-29T05:27:44.340 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0: { 2024-03-29T05:27:44.340 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-29T05:27:44.340 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:44.353 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1: { 2024-03-29T05:27:44.353 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-29T05:27:44.353 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:44.369 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2: { 2024-03-29T05:27:44.369 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-29T05:27:44.369 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:44.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-03-29T05:27:44.514 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0: { 2024-03-29T05:27:44.515 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-29T05:27:44.515 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:44.527 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1: { 2024-03-29T05:27:44.528 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-29T05:27:44.528 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:44.540 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2: { 2024-03-29T05:27:44.540 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-29T05:27:44.540 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-03-29T05:27:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-03-29T05:27:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-03-29T05:27:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-03-29T05:27:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-03-29T05:27:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-29T05:27:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-29T05:27:45.017 INFO:tasks.workunit.client.0.smithi123.stdout:created pool test snap snap01 2024-03-29T05:27:45.248 INFO:tasks.workunit.client.0.smithi123.stdout:created pool test snap snap02 2024-03-29T05:27:45.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-03-29T05:27:45.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-29T05:27:45.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-29T05:27:45.695 INFO:tasks.workunit.client.0.smithi123.stdout:created pool test snap snap13 2024-03-29T05:27:45.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-03-29T05:27:45.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-29T05:27:45.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-29T05:27:46.231 INFO:tasks.workunit.client.0.smithi123.stdout:created pool test snap snap24 2024-03-29T05:27:46.470 INFO:tasks.workunit.client.0.smithi123.stdout:created pool test snap snap25 2024-03-29T05:27:46.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 5414 2024-03-29T05:27:46.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-03-29T05:27:46.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-29T05:27:46.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-03-29T05:27:47.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-03-29T05:27:47.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-29T05:27:47.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-29T05:27:47.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-03-29T05:27:47.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-03-29T05:27:47.482 INFO:tasks.workunit.client.0.smithi123.stdout:pgid is 1.3 (primary: osd.1) 2024-03-29T05:27:47.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-03-29T05:27:47.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-03-29T05:27:47.482 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-03-29T05:27:47.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-03-29T05:27:47.825 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs_brief 2024-03-29T05:27:47.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-03-29T05:27:47.844 INFO:tasks.workunit.client.0.smithi123.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-03-29T05:27:47.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-03-29T05:27:47.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-03-29T05:27:47.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T05:27:47.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T05:27:47.845 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:27:47.845 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:27:47.845 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:27:47.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20229/ceph-osd.1.asok 2024-03-29T05:27:47.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-03-29T05:27:47.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.20229/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-03-29T05:27:47.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:27:47.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-29T05:27:48.005 INFO:tasks.workunit.client.0.smithi123.stdout:{"success":true}{ 2024-03-29T05:27:48.005 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "objxxx", 2024-03-29T05:27:48.005 INFO:tasks.workunit.client.0.smithi123.stdout: "clones": [ 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 2, 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [ 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 1, 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap01" 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 2, 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap02" 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 6, 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: "overlaps": [] 2024-03-29T05:27:48.006 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 3, 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [ 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 3, 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap13" 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 6, 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "overlaps": [] 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 5, 2024-03-29T05:27:48.007 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [ 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 4, 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap24" 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 5, 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap25" 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 6, 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: "overlaps": [] 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:27:48.008 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:27:48.009 INFO:tasks.workunit.client.0.smithi123.stdout: "id": "head", 2024-03-29T05:27:48.009 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [], 2024-03-29T05:27:48.009 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 5 2024-03-29T05:27:48.009 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:27:48.009 INFO:tasks.workunit.client.0.smithi123.stdout: ] 2024-03-29T05:27:48.009 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:48.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-29T05:27:48.125 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:27:48.125 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": true, 2024-03-29T05:27:48.126 INFO:tasks.workunit.client.0.smithi123.stdout: "must": true, 2024-03-29T05:27:48.126 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "0.000000" 2024-03-29T05:27:48.126 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:27:48.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-03-29T05:27:51.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-29T05:27:51.432 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:27:51.432 INFO:tasks.workunit.client.0.smithi123.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-29T05:27:37.078815+0000 26'7 26:33 [1,2,0] 1 [1,2,0] 1 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.432 INFO:tasks.workunit.client.0.smithi123.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:27:37.781327+0000 0'0 26:29 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.432 INFO:tasks.workunit.client.0.smithi123.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:27:37.078758+0000 0'0 21:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.432 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:27:37.078783+0000 0'0 26:29 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.432 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:51.432 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:27:51.433 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:27:51.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-03-29T05:27:51.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-29T05:27:51.736 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:27:51.737 INFO:tasks.workunit.client.0.smithi123.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-29T05:27:37.078815+0000 26'7 26:33 [1,2,0] 1 [1,2,0] 1 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.737 INFO:tasks.workunit.client.0.smithi123.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:27:37.781327+0000 0'0 26:29 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.737 INFO:tasks.workunit.client.0.smithi123.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:27:37.078758+0000 0'0 21:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.737 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:27:37.078783+0000 0'0 26:29 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:27:51.737 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:51.737 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:27:51.738 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:27:51.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-03-29T05:27:52.203 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T05:27:52.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-03-29T05:27:52.814 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T05:27:52.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-03-29T05:27:57.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-03-29T05:27:57.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-29T05:27:57.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:27:57.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:27:57.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:27:57.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:27:57.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:27:58.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:27:58.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:27:58.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-29T05:27:58.181 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.176+0000 7f35a7ab6ac0 1 bdev(0x5597da83c000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-29T05:27:58.181 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.176+0000 7f35a7ab6ac0 1 bdev(0x5597da83c000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:27:58.181 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.176+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-29T05:27:58.181 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.176+0000 7f35a7ab6ac0 1 bdev(0x5597da83c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-29T05:27:58.181 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.176+0000 7f35a7ab6ac0 1 bdev(0x5597da83c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:27:58.182 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.176+0000 7f35a7ab6ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-29T05:27:58.182 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.176+0000 7f35a7ab6ac0 1 bluefs mount 2024-03-29T05:27:58.200 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.196+0000 7f35a7ab6ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:27:58.226 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.220+0000 7f35a7ab6ac0 1 bluefs mount shared_bdev_used = 0 2024-03-29T05:27:58.226 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.220+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:27:58.229 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:27:58.229 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:27:58.229 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:27:58.229 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:27:58.229 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:27:58.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:27:58.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:27:58.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:27:58.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:27:58.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:27:58.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:27:58.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.224+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:27:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:27:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2024-03-29T05:27:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-29T05:27:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-29T05:27:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-29T05:27:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 freelist init 2024-03-29T05:27:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 freelist _read_cfg 2024-03-29T05:27:58.279 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.77383e-07 2024-03-29T05:27:58.279 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bluefs umount 2024-03-29T05:27:58.279 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.272+0000 7f35a7ab6ac0 1 bdev(0x5597da83c700 td/osd-mapper/0/block) close 2024-03-29T05:27:58.476 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.472+0000 7f35a7ab6ac0 1 bdev(0x5597da83c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-29T05:27:58.477 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.472+0000 7f35a7ab6ac0 1 bdev(0x5597da83c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:27:58.477 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.472+0000 7f35a7ab6ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-29T05:27:58.477 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.472+0000 7f35a7ab6ac0 1 bluefs mount 2024-03-29T05:27:58.478 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.472+0000 7f35a7ab6ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:27:58.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.480+0000 7f35a7ab6ac0 1 bluefs mount shared_bdev_used = 27459584 2024-03-29T05:27:58.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.480+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:27:58.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:27:58.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:27:58.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:27:58.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:27:58.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:27:58.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:27:58.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:27:58.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:27:58.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:27:58.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:27:58.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:27:58.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:58.484+0000 7f35a7ab6ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:27:59.640 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.636+0000 7f35a7ab6ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:27:59.641 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 0b 05 58 02 00 00 fc 6e 84 1e ff 7b 40 91 a7 eb |..X....n...{@...| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 7c 23 fd 30 46 ae 00 00 00 00 1c 00 00 00 00 00 ||#.0F...........| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:* 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-29T05:27:59.643 INFO:tasks.workunit.client.0.smithi123.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-29T05:27:59.644 INFO:tasks.workunit.client.0.smithi123.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 c2 0f 61 0e 1e 90 |re..........a...| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:00000230 40 3d a7 01 4f 1b 3b 6a 04 bf 00 00 00 00 00 00 |@=..O.;j........| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:00000240 00 00 1c 00 00 00 f9 50 06 66 8a 06 cf 2a 01 00 |.......P.f...*..| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:0000025e 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.645 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:27:59.646 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:0000003f 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:27:59.647 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:27:59.648 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:27:59.649 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.650 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-29T05:27:59.650 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.650 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-29T05:27:59.650 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:27:59.650 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.636+0000 7f35a7ab6ac0 1 bluefs umount 2024-03-29T05:27:59.650 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.636+0000 7f35a7ab6ac0 1 bdev(0x5597da83c700 td/osd-mapper/0/block) close 2024-03-29T05:27:59.905 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.900+0000 7f35a7ab6ac0 1 freelist shutdown 2024-03-29T05:27:59.905 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.900+0000 7f35a7ab6ac0 1 bdev(0x5597da83c000 td/osd-mapper/0/block) close 2024-03-29T05:27:59.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:27:59.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-29T05:27:59.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-03-29T05:27:59.996 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.988+0000 7f740a088ac0 1 bdev(0x55e5457c8000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-29T05:27:59.997 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.992+0000 7f740a088ac0 1 bdev(0x55e5457c8000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:27:59.997 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.992+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-29T05:27:59.997 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.992+0000 7f740a088ac0 1 bdev(0x55e5457c8700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-29T05:27:59.997 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.992+0000 7f740a088ac0 1 bdev(0x55e5457c8700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:27:59.997 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.992+0000 7f740a088ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-29T05:27:59.997 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:27:59.992+0000 7f740a088ac0 1 bluefs mount 2024-03-29T05:28:00.008 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.004+0000 7f740a088ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:00.017 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 1 bluefs mount shared_bdev_used = 0 2024-03-29T05:28:00.017 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:00.021 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:00.022 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:00.022 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.012+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:00.066 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:00.067 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-03-29T05:28:00.067 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-29T05:28:00.067 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-29T05:28:00.067 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-29T05:28:00.067 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 freelist init 2024-03-29T05:28:00.067 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 freelist _read_cfg 2024-03-29T05:28:00.068 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-03-29T05:28:00.068 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bluefs umount 2024-03-29T05:28:00.068 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.060+0000 7f740a088ac0 1 bdev(0x55e5457c8700 td/osd-mapper/2/block) close 2024-03-29T05:28:00.276 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.272+0000 7f740a088ac0 1 bdev(0x55e5457c8700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-29T05:28:00.277 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.272+0000 7f740a088ac0 1 bdev(0x55e5457c8700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:00.277 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.272+0000 7f740a088ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-29T05:28:00.277 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.272+0000 7f740a088ac0 1 bluefs mount 2024-03-29T05:28:00.277 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.272+0000 7f740a088ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:00.283 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.276+0000 7f740a088ac0 1 bluefs mount shared_bdev_used = 27459584 2024-03-29T05:28:00.284 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.276+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:00.288 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:00.288 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:00.280+0000 7f740a088ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:01.322 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:01.316+0000 7f740a088ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:01.324 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:01.316+0000 7f740a088ac0 1 bluefs umount 2024-03-29T05:28:01.324 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:01.316+0000 7f740a088ac0 1 bdev(0x55e5457c8700 td/osd-mapper/2/block) close 2024-03-29T05:28:01.568 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:01.564+0000 7f740a088ac0 1 freelist shutdown 2024-03-29T05:28:01.568 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:01.564+0000 7f740a088ac0 1 bdev(0x55e5457c8000 td/osd-mapper/2/block) close 2024-03-29T05:28:01.786 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:01.786 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:01.786 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:01.786 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:01.786 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:01.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:28:01.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-03-29T05:28:01.788 INFO:tasks.workunit.client.0.smithi123.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-03-29T05:28:01.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:28:01.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-29T05:28:02.408 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-29T05:28:02.408 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 0b 05 58 02 00 00 fc 6e 84 1e ff 7b 40 91 a7 eb |..X....n...{@...| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 7c 23 fd 30 46 ae 02 00 00 00 1c 00 00 00 00 00 ||#.0F...........| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:* 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-29T05:28:02.409 INFO:tasks.workunit.client.0.smithi123.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-29T05:28:02.410 INFO:tasks.workunit.client.0.smithi123.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 bd 92 25 23 73 1c |re..........%#s.| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:00000230 49 54 89 b2 d8 22 7a 45 a7 d3 00 00 00 00 00 00 |IT..."zE........| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:00000240 00 00 1c 00 00 00 46 51 06 66 d9 29 c2 1e 01 00 |......FQ.f.)....| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:0000025e 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.411 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout:0000003f 2024-03-29T05:28:02.412 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:02.413 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.414 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.415 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-29T05:28:02.416 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.416 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-29T05:28:02.416 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:02.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-29T05:28:02.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-29T05:28:02.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-03-29T05:28:02.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-29T05:28:04.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-03-29T05:28:04.698 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-03-29T05:28:04.700 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-03-29T05:28:04.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:04.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-03-29T05:28:04.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-03-29T05:28:04.701 INFO:tasks.workunit.client.0.smithi123.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-03-29T05:28:04.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-29T05:28:04.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-29T05:28:04.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.724+0000 7f649354aac0 1 bdev(0x564d9418c000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-29T05:28:04.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.724+0000 7f649354aac0 1 bdev(0x564d9418c000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:04.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.724+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-29T05:28:04.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.724+0000 7f649354aac0 1 bdev(0x564d9418c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-29T05:28:04.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.724+0000 7f649354aac0 1 bdev(0x564d9418c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:04.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.724+0000 7f649354aac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-29T05:28:04.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.724+0000 7f649354aac0 1 bluefs mount 2024-03-29T05:28:04.741 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.736+0000 7f649354aac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:04.750 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.744+0000 7f649354aac0 1 bluefs mount shared_bdev_used = 0 2024-03-29T05:28:04.751 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.744+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:04.754 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:04.754 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:04.754 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:04.754 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:04.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.748+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.752+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.752+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2051 2024-03-29T05:28:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.752+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-29T05:28:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.752+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-29T05:28:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.752+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-29T05:28:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.756+0000 7f649354aac0 1 freelist init 2024-03-29T05:28:04.761 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.756+0000 7f649354aac0 1 freelist _read_cfg 2024-03-29T05:28:04.761 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.756+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.77383e-07 2024-03-29T05:28:04.761 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.756+0000 7f649354aac0 1 bluefs umount 2024-03-29T05:28:04.761 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:04.756+0000 7f649354aac0 1 bdev(0x564d9418c700 td/osd-mapper/0/block) close 2024-03-29T05:28:05.008 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.004+0000 7f649354aac0 1 bdev(0x564d9418c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-29T05:28:05.009 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.004+0000 7f649354aac0 1 bdev(0x564d9418c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:05.009 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.004+0000 7f649354aac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-29T05:28:05.009 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.004+0000 7f649354aac0 1 bluefs mount 2024-03-29T05:28:05.024 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.020+0000 7f649354aac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:05.034 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.028+0000 7f649354aac0 1 bluefs mount shared_bdev_used = 9568256 2024-03-29T05:28:05.034 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.028+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:05.037 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:05.037 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:05.037 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:05.037 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:05.037 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:05.037 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.032+0000 7f649354aac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:05.387 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.384+0000 7f649354aac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:05.389 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 0b 05 58 02 00 00 fc 6e 84 1e ff 7b 40 91 a7 eb |..X....n...{@...| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 7c 23 fd 30 46 ae 00 00 00 00 1c 00 00 00 00 00 ||#.0F...........| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:* 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-29T05:28:05.390 INFO:tasks.workunit.client.0.smithi123.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-29T05:28:05.391 INFO:tasks.workunit.client.0.smithi123.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 c2 0f 61 0e 1e 90 |re..........a...| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:00000230 40 3d a7 01 4f 1b 3b 6a 04 bf 00 00 00 00 00 00 |@=..O.;j........| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:00000240 00 00 1c 00 00 00 f9 50 06 66 8a 06 cf 2a 01 00 |.......P.f...*..| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout:0000025e 2024-03-29T05:28:05.392 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:05.393 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:0000003f 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:05.394 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:05.395 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:05.396 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:05.397 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.384+0000 7f649354aac0 1 bluefs umount 2024-03-29T05:28:05.398 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.384+0000 7f649354aac0 1 bdev(0x564d9418c700 td/osd-mapper/0/block) close 2024-03-29T05:28:05.632 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.628+0000 7f649354aac0 1 freelist shutdown 2024-03-29T05:28:05.632 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:05.628+0000 7f649354aac0 1 bdev(0x564d9418c000 td/osd-mapper/0/block) close 2024-03-29T05:28:05.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-29T05:28:05.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-29T05:28:05.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-29T05:28:06.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:06.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:06.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-29T05:28:06.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-29T05:28:06.836 INFO:tasks.workunit.client.0.smithi123.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:06.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-29T05:28:06.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.fuIkSvGzMQ_the_val 2024-03-29T05:28:06.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:06.839 INFO:tasks.workunit.client.0.smithi123.stdout:Value dumped in: /tmp/tmp.fuIkSvGzMQ_the_val 2024-03-29T05:28:06.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.fuIkSvGzMQ_the_val' 2024-03-29T05:28:06.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.fuIkSvGzMQ_the_val 2024-03-29T05:28:07.414 INFO:tasks.workunit.client.0.smithi123.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-29T05:28:07.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:28:07.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.fuIkSvGzMQ_the_val 2024-03-29T05:28:07.916 INFO:tasks.workunit.client.0.smithi123.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-29T05:28:07.916 INFO:tasks.workunit.client.0.smithi123.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-29T05:28:07.916 INFO:tasks.workunit.client.0.smithi123.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-29T05:28:07.916 INFO:tasks.workunit.client.0.smithi123.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-29T05:28:07.916 INFO:tasks.workunit.client.0.smithi123.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-29T05:28:07.917 INFO:tasks.workunit.client.0.smithi123.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-03-29T05:28:07.917 INFO:tasks.workunit.client.0.smithi123.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-29T05:28:07.917 INFO:tasks.workunit.client.0.smithi123.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-29T05:28:07.917 INFO:tasks.workunit.client.0.smithi123.stdout:0000073 2024-03-29T05:28:07.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-29T05:28:07.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:08.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.fuIkSvGzMQ_the_val 2024-03-29T05:28:10.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:10.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:10.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-29T05:28:10.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-03-29T05:28:10.314 INFO:tasks.workunit.client.0.smithi123.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-03-29T05:28:10.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-29T05:28:10.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-29T05:28:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.336+0000 7f20774f0ac0 1 bdev(0x55cd2645c000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-29T05:28:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.336+0000 7f20774f0ac0 1 bdev(0x55cd2645c000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.336+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-29T05:28:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.336+0000 7f20774f0ac0 1 bdev(0x55cd2645c700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-29T05:28:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.336+0000 7f20774f0ac0 1 bdev(0x55cd2645c700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.336+0000 7f20774f0ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-29T05:28:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.336+0000 7f20774f0ac0 1 bluefs mount 2024-03-29T05:28:10.353 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.348+0000 7f20774f0ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:10.363 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.356+0000 7f20774f0ac0 1 bluefs mount shared_bdev_used = 0 2024-03-29T05:28:10.363 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.356+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:10.366 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:10.366 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:10.366 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:10.366 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:10.367 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.360+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.364+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.364+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-03-29T05:28:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.364+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-03-29T05:28:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.364+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-29T05:28:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.364+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-03-29T05:28:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.364+0000 7f20774f0ac0 1 freelist init 2024-03-29T05:28:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.364+0000 7f20774f0ac0 1 freelist _read_cfg 2024-03-29T05:28:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.368+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.72206e-07 2024-03-29T05:28:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.368+0000 7f20774f0ac0 1 bluefs umount 2024-03-29T05:28:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.368+0000 7f20774f0ac0 1 bdev(0x55cd2645c700 td/osd-mapper/1/block) close 2024-03-29T05:28:10.624 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.620+0000 7f20774f0ac0 1 bdev(0x55cd2645c700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-29T05:28:10.625 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.620+0000 7f20774f0ac0 1 bdev(0x55cd2645c700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:10.625 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.620+0000 7f20774f0ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-29T05:28:10.625 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.620+0000 7f20774f0ac0 1 bluefs mount 2024-03-29T05:28:10.628 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.620+0000 7f20774f0ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:10.638 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.632+0000 7f20774f0ac0 1 bluefs mount shared_bdev_used = 9568256 2024-03-29T05:28:10.638 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.632+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:10.641 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:10.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:10.636+0000 7f20774f0ac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:11.045 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:11.040+0000 7f20774f0ac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 0b 05 58 02 00 00 fc 6e 84 1e ff 7b 40 91 a7 eb |..X....n...{@...| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 7c 23 fd 30 46 ae 01 00 00 00 1c 00 00 00 00 00 ||#.0F...........| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:* 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-29T05:28:11.047 INFO:tasks.workunit.client.0.smithi123.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-29T05:28:11.048 INFO:tasks.workunit.client.0.smithi123.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:00000220 72 65 1c 00 00 00 0a 00 00 00 74 e4 13 64 61 f8 |re........t..da.| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:00000230 43 6f a7 2f 9a 43 b5 ce 27 57 00 00 00 00 00 00 |Co./.C..'W......| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:00000240 00 00 1c 00 00 00 20 51 06 66 6f 88 07 09 01 00 |...... Q.fo.....| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:0000025e 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-29T05:28:11.049 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-29T05:28:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:0000003f 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:11.051 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:11.052 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:11.053 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:11.054 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:11.040+0000 7f20774f0ac0 1 bluefs umount 2024-03-29T05:28:11.055 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:11.040+0000 7f20774f0ac0 1 bdev(0x55cd2645c700 td/osd-mapper/1/block) close 2024-03-29T05:28:11.292 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:11.288+0000 7f20774f0ac0 1 freelist shutdown 2024-03-29T05:28:11.292 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:11.288+0000 7f20774f0ac0 1 bdev(0x55cd2645c000 td/osd-mapper/1/block) close 2024-03-29T05:28:11.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-29T05:28:11.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-29T05:28:11.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-29T05:28:12.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:12.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:12.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-29T05:28:12.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-29T05:28:12.456 INFO:tasks.workunit.client.0.smithi123.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-29T05:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.6PBLiE9Rcx_the_val 2024-03-29T05:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.6PBLiE9Rcx_the_val' 2024-03-29T05:28:12.459 INFO:tasks.workunit.client.0.smithi123.stdout:Value dumped in: /tmp/tmp.6PBLiE9Rcx_the_val 2024-03-29T05:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.6PBLiE9Rcx_the_val 2024-03-29T05:28:13.181 INFO:tasks.workunit.client.0.smithi123.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-29T05:28:13.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:28:13.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.6PBLiE9Rcx_the_val 2024-03-29T05:28:13.534 INFO:tasks.workunit.client.0.smithi123.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-29T05:28:13.534 INFO:tasks.workunit.client.0.smithi123.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-29T05:28:13.534 INFO:tasks.workunit.client.0.smithi123.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-29T05:28:13.534 INFO:tasks.workunit.client.0.smithi123.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-29T05:28:13.534 INFO:tasks.workunit.client.0.smithi123.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-29T05:28:13.535 INFO:tasks.workunit.client.0.smithi123.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-03-29T05:28:13.535 INFO:tasks.workunit.client.0.smithi123.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-29T05:28:13.535 INFO:tasks.workunit.client.0.smithi123.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-29T05:28:13.535 INFO:tasks.workunit.client.0.smithi123.stdout:0000073 2024-03-29T05:28:13.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-29T05:28:13.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:14.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.6PBLiE9Rcx_the_val 2024-03-29T05:28:15.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:15.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:15.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-03-29T05:28:15.922 INFO:tasks.workunit.client.0.smithi123.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-03-29T05:28:15.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-03-29T05:28:15.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-29T05:28:15.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-29T05:28:15.952 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.943+0000 7fda5964dac0 1 bdev(0x55d2c36d0000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-29T05:28:15.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.947+0000 7fda5964dac0 1 bdev(0x55d2c36d0000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:15.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.947+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-29T05:28:15.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.947+0000 7fda5964dac0 1 bdev(0x55d2c36d0700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-29T05:28:15.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.947+0000 7fda5964dac0 1 bdev(0x55d2c36d0700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:15.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.947+0000 7fda5964dac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-29T05:28:15.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.947+0000 7fda5964dac0 1 bluefs mount 2024-03-29T05:28:15.972 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.967+0000 7fda5964dac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:15.982 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.975+0000 7fda5964dac0 1 bluefs mount shared_bdev_used = 0 2024-03-29T05:28:15.983 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.975+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:15.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:15.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:15.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:15.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:15.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:15.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:15.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:15.987 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:15.987 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:15.987 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:15.987 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:15.987 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.979+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:15.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.983+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:15.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.983+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-03-29T05:28:15.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.983+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-29T05:28:15.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.983+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-29T05:28:15.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.983+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-29T05:28:15.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.983+0000 7fda5964dac0 1 freelist init 2024-03-29T05:28:15.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.983+0000 7fda5964dac0 1 freelist _read_cfg 2024-03-29T05:28:15.993 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.987+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-03-29T05:28:15.993 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.987+0000 7fda5964dac0 1 bluefs umount 2024-03-29T05:28:15.993 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:15.987+0000 7fda5964dac0 1 bdev(0x55d2c36d0700 td/osd-mapper/2/block) close 2024-03-29T05:28:16.240 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.235+0000 7fda5964dac0 1 bdev(0x55d2c36d0700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-29T05:28:16.241 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.235+0000 7fda5964dac0 1 bdev(0x55d2c36d0700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-29T05:28:16.241 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.235+0000 7fda5964dac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-29T05:28:16.241 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.235+0000 7fda5964dac0 1 bluefs mount 2024-03-29T05:28:16.256 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.247+0000 7fda5964dac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-29T05:28:16.266 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.259+0000 7fda5964dac0 1 bluefs mount shared_bdev_used = 9568256 2024-03-29T05:28:16.266 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.259+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-29T05:28:16.269 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-29T05:28:16.269 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-29T05:28:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.263+0000 7fda5964dac0 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-29T05:28:16.544 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.539+0000 7fda5964dac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-29T05:28:16.545 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 0b 05 58 02 00 00 fc 6e 84 1e ff 7b 40 91 a7 eb |..X....n...{@...| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 7c 23 fd 30 46 ae 02 00 00 00 1c 00 00 00 00 00 ||#.0F...........| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:* 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-29T05:28:16.546 INFO:tasks.workunit.client.0.smithi123.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-29T05:28:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 bd 92 25 23 73 1c |re..........%#s.| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:00000230 49 54 89 b2 d8 22 7a 45 a7 d3 00 00 00 00 00 00 |IT..."zE........| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:00000240 00 00 1c 00 00 00 46 51 06 66 d9 29 c2 1e 01 00 |......FQ.f.)....| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:0000025e 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-29T05:28:16.548 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.549 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:16.549 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:16.549 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:16.549 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:16.549 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-29T05:28:16.549 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:0000003f 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-29T05:28:16.550 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000047 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:16.551 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:16.552 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout:0000003b 2024-03-29T05:28:16.553 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.554 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-29T05:28:16.554 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.554 INFO:tasks.workunit.client.0.smithi123.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-29T05:28:16.554 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:16.554 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.539+0000 7fda5964dac0 1 bluefs umount 2024-03-29T05:28:16.554 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.539+0000 7fda5964dac0 1 bdev(0x55d2c36d0700 td/osd-mapper/2/block) close 2024-03-29T05:28:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.783+0000 7fda5964dac0 1 freelist shutdown 2024-03-29T05:28:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:16.783+0000 7fda5964dac0 1 bdev(0x55d2c36d0000 td/osd-mapper/2/block) close 2024-03-29T05:28:16.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-29T05:28:16.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-29T05:28:16.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-29T05:28:18.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:18.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:18.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-29T05:28:18.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-29T05:28:18.060 INFO:tasks.workunit.client.0.smithi123.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:18.061 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-29T05:28:18.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.MBhoBtE68o_the_val 2024-03-29T05:28:18.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:18.063 INFO:tasks.workunit.client.0.smithi123.stdout:Value dumped in: /tmp/tmp.MBhoBtE68o_the_val 2024-03-29T05:28:18.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.MBhoBtE68o_the_val' 2024-03-29T05:28:18.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.MBhoBtE68o_the_val 2024-03-29T05:28:18.613 INFO:tasks.workunit.client.0.smithi123.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-29T05:28:19.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:28:19.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.MBhoBtE68o_the_val 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-29T05:28:19.131 INFO:tasks.workunit.client.0.smithi123.stdout:0000073 2024-03-29T05:28:19.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-29T05:28:19.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:20.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.MBhoBtE68o_the_val 2024-03-29T05:28:21.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:21.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-03-29T05:28:21.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-03-29T05:28:21.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-29T05:28:21.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:28:21.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-03-29T05:28:21.523 INFO:tasks.workunit.client.0.smithi123.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-03-29T05:28:21.524 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-03-29T05:28:21.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-03-29T05:28:21.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:21.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-29T05:28:21.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-03-29T05:28:21.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:28:21.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-29T05:28:21.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:28:21.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:28:21.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:28:21.530 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:28:21.530 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:21.530 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20229/$cluster-$name.asok' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20229/$cluster-$name.asok' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:28:21.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-03-29T05:28:21.532 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:28:21.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:28:21.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20229/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:28:21.532 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-03-29T05:28:21.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:28:21.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:28:21.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:28:21.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:28:21.584 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:21.579+0000 7f5b599897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:21.587 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:21.579+0000 7f5b599897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:21.588 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:21.583+0000 7f5b599897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:28:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:28:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:28:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:28:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:28:21.887 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:28:21.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:21.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:28:21.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:21.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:28:22.123 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:22.115+0000 7f5b599897c0 -1 Falling back to public interface 2024-03-29T05:28:22.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:23.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:23.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:23.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:28:23.231 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:28:23.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:23.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:28:23.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:24.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:24.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:24.548 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:28:24.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:28:24.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:24.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:28:24.635 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:24.627+0000 7f5b599897c0 -1 osd.0 28 log_to_monitors true 2024-03-29T05:28:24.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:25.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:25.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:25.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:28:25.877 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:28:25.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:25.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:28:26.212 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 31 up_thru 31 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6800/3398897694,v1:127.0.0.1:6801/3398897694] [v2:127.0.0.1:6802/3398897694,v1:127.0.0.1:6803/3398897694] exists,up c20f610e-1e90-403d-a701-4f1b3b6a04bf 2024-03-29T05:28:26.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:28:26.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:28:26.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:28:26.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:26.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-29T05:28:26.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:28:26.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:28:26.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20229/$cluster-$name.asok' 2024-03-29T05:28:26.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20229/$cluster-$name.asok' 2024-03-29T05:28:26.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:28:26.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-29T05:28:26.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-03-29T05:28:26.218 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:28:26.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:28:26.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20229/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:28:26.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-03-29T05:28:26.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:28:26.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:28:26.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:28:26.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:28:26.258 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:26.251+0000 7f26c8e097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:26.269 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:26.263+0000 7f26c8e097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:26.271 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:26.263+0000 7f26c8e097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:26.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:28:26.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:28:26.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:28:26.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:28:26.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:28:26.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:26.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:28:26.533 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:28:26.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:26.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:28:26.807 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:26.799+0000 7f26c8e097c0 -1 Falling back to public interface 2024-03-29T05:28:26.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:27.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:27.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:27.883 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:28:27.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:28:27.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:27.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:28:28.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:29.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:29.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:29.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:28:29.219 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:28:29.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:29.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:28:29.391 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:29.383+0000 7f26c8e097c0 -1 osd.1 28 log_to_monitors true 2024-03-29T05:28:29.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:30.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:30.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:30.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:28:30.650 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:28:30.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:30.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:28:31.040 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3520084914,v1:127.0.0.1:6811/3520084914] [v2:127.0.0.1:6812/3520084914,v1:127.0.0.1:6813/3520084914] exists,up 74e41364-61f8-436f-a72f-9a43b5ce2757 2024-03-29T05:28:31.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:28:31.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:28:31.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:28:31.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:31.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:28:31.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20229/$cluster-$name.asok' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20229/$cluster-$name.asok' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:28:31.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:28:31.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:28:31.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:28:31.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-03-29T05:28:31.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T05:28:31.050 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:28:31.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=fc6e841e-ff7b-4091-a7eb-7c23fd3046ae --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.20229/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:28:31.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-03-29T05:28:31.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T05:28:31.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:28:31.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:28:31.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:28:31.093 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:31.087+0000 7f6e64c5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:31.117 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:31.111+0000 7f6e64c5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:31.119 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:31.111+0000 7f6e64c5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:28:31.366 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:28:31.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:31.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:28:31.670 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:31.663+0000 7f6e64c5c7c0 -1 Falling back to public interface 2024-03-29T05:28:31.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:32.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:32.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:32.712 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:28:32.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:28:32.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:32.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:28:33.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:34.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:34.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:34.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:28:34.083 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:28:34.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:34.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:28:34.320 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:34.311+0000 7f6e64c5c7c0 -1 osd.2 28 log_to_monitors true 2024-03-29T05:28:34.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:28:35.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:28:35.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:28:35.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:28:35.451 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:28:35.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:28:35.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:28:35.949 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 38 up_thru 38 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/2357037515,v1:127.0.0.1:6819/2357037515] [v2:127.0.0.1:6820/2357037515,v1:127.0.0.1:6821/2357037515] exists,up bd922523-731c-4954-89b2-d8227a45a7d3 2024-03-29T05:28:35.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:28:35.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:28:35.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:28:35.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-03-29T05:28:36.953 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-03-29T05:28:36.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-03-29T05:28:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-03-29T05:28:37.097 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:28:37.097 INFO:tasks.workunit.client.0.smithi123.stdout: "version": "19.0.0-1959-g8da23624", 2024-03-29T05:28:37.097 INFO:tasks.workunit.client.0.smithi123.stdout: "release": "squid", 2024-03-29T05:28:37.097 INFO:tasks.workunit.client.0.smithi123.stdout: "release_type": "dev" 2024-03-29T05:28:37.097 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:28:37.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:37.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-03-29T05:28:37.240 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:28:37.240 INFO:tasks.workunit.client.0.smithi123.stdout: "version": "19.0.0-1959-g8da23624", 2024-03-29T05:28:37.240 INFO:tasks.workunit.client.0.smithi123.stdout: "release": "squid", 2024-03-29T05:28:37.240 INFO:tasks.workunit.client.0.smithi123.stdout: "release_type": "dev" 2024-03-29T05:28:37.240 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:28:37.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-29T05:28:37.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-03-29T05:28:37.380 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:28:37.380 INFO:tasks.workunit.client.0.smithi123.stdout: "version": "19.0.0-1959-g8da23624", 2024-03-29T05:28:37.380 INFO:tasks.workunit.client.0.smithi123.stdout: "release": "squid", 2024-03-29T05:28:37.380 INFO:tasks.workunit.client.0.smithi123.stdout: "release_type": "dev" 2024-03-29T05:28:37.380 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:28:37.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-29T05:28:37.421 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "objxxx", 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: "clones": [ 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 2, 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [ 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 1, 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap01" 2024-03-29T05:28:37.422 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 2, 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap02" 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 6, 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: "overlaps": [] 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 3, 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [ 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.423 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 3, 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap13" 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 6, 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: "overlaps": [] 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 5, 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [ 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 4, 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap24" 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:28:37.424 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: "id": 5, 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: "name": "snap25" 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 6, 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: "overlaps": [] 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: { 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: "id": "head", 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: "snapshots": [], 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: "size": 5 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: } 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout: ] 2024-03-29T05:28:37.425 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:28:37.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-03-29T05:28:37.826 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is unset 2024-03-29T05:28:37.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-03-29T05:28:38.354 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is unset 2024-03-29T05:28:38.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-29T05:28:38.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-29T05:28:38.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-03-29T05:28:38.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-29T05:28:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:28:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:1.3 4 0 4 0 0 23 0 0 7 0 7 active+undersized+degraded 2024-03-29T05:28:30.896545+0000 26'7 37:54 [1,0] 1 [1,0] 1 26'7 2024-03-29T05:27:48.533569+0000 26'7 2024-03-29T05:27:48.533569+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:27:48.533569+0000 4 0 2024-03-29T05:28:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-29T05:28:31.039348+0000 0'0 37:30 [0,1] 0 [0,1] 0 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-29T05:28:31.039244+0000 0'0 37:20 [0,1] 0 [0,1] 0 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:38.939 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-29T05:28:30.793153+0000 0'0 37:22 [1,0] 1 [1,0] 1 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:38.939 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:38.940 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:28:38.940 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:28:38.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-03-29T05:28:40.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-29T05:28:41.091 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:28:41.091 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": true, 2024-03-29T05:28:41.091 INFO:tasks.workunit.client.0.smithi123.stdout: "must": true, 2024-03-29T05:28:41.092 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "0.000000" 2024-03-29T05:28:41.092 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:28:41.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-03-29T05:28:46.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-29T05:28:41.625550+0000 26'7 42:79 [1,2,0] 1 [1,2,0] 1 26'7 2024-03-29T05:28:41.625215+0000 26'7 2024-03-29T05:28:41.625215+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:28:41.625215+0000 4 0 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:28:35.793839+0000 0'0 42:46 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:28:36.248423+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:28:36.248294+0000 0'0 42:37 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:28:46.396 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:28:46.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:46.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-29T05:28:46.413 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:28:41.619+0000 7f26ab590640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-29T05:28:46.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-03-29T05:28:46.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-03-29T05:28:46.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-29T05:28:46.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-03-29T05:28:46.421 INFO:tasks.workunit.client.0.smithi123.stdout:prev count: 1 2024-03-29T05:28:46.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-03-29T05:28:46.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-29T05:28:46.549 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:28:46.549 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": true, 2024-03-29T05:28:46.549 INFO:tasks.workunit.client.0.smithi123.stdout: "must": true, 2024-03-29T05:28:46.549 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "0.000000" 2024-03-29T05:28:46.549 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:28:46.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-03-29T05:28:51.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-29T05:28:51.851 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:28:51.851 INFO:tasks.workunit.client.0.smithi123.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-29T05:28:47.498419+0000 26'7 42:88 [1,2,0] 1 [1,2,0] 1 26'7 2024-03-29T05:28:47.498206+0000 26'7 2024-03-29T05:28:47.498206+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:28:47.498206+0000 4 0 2024-03-29T05:28:51.851 INFO:tasks.workunit.client.0.smithi123.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:28:35.793839+0000 0'0 42:46 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:51.851 INFO:tasks.workunit.client.0.smithi123.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:28:36.248423+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:51.851 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:28:36.248294+0000 0'0 42:37 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-29T05:27:35.795284+0000 0'0 2024-03-29T05:27:35.795284+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:27:35.795284+0000 0 0 2024-03-29T05:28:51.851 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:28:51.851 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:28:51.852 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:28:51.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:51.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-29T05:28:51.870 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:28:41.619+0000 7f26ab590640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-29T05:28:51.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-29T05:28:51.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-03-29T05:28:51.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-03-29T05:28:51.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-29T05:28:51.874 INFO:tasks.workunit.client.0.smithi123.stdout:current count: 1 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:28:51.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:28:52.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:28:52.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-29T05:28:52.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-29T05:28:52.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-29T05:28:52.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-03-29T05:28:52.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-29T05:28:53.754 INFO:tasks.workunit.client.0.smithi123.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:53.754 INFO:tasks.workunit.client.0.smithi123.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-29T05:28:53.754 INFO:tasks.workunit.client.0.smithi123.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-29T05:28:53.754 INFO:tasks.workunit.client.0.smithi123.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-29T05:28:53.754 INFO:tasks.workunit.client.0.smithi123.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:53.754 INFO:tasks.workunit.client.0.smithi123.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-29T05:28:53.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-29T05:28:53.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-29T05:28:53.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-03-29T05:28:53.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-29T05:28:54.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-03-29T05:28:54.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-03-29T05:28:54.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-03-29T05:28:54.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-03-29T05:28:54.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-29T05:28:54.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:28:54.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-29T05:28:54.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:28:54.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:28:54.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:28:54.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:28:54.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:28:54.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:28:54.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:28:54.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:28:54.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:28:54.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:28:54.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:28:54.950 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:28:54.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:28:54.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:28:54.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:28:54.951 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:28:54.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:28:54.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:28:54.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-29T05:28:54.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:28:54.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:54.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:28:54.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20229 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:28:54.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:28:54.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:28:54.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:28:54.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:28:54.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:28:54.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:28:54.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:28:54.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:28:54.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:28:54.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:28:54.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:28:54.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:28:54.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:28:54.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-29T05:28:54.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-29T05:28:54.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:28:54.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:54.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20229 2024-03-29T05:28:54.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20229 2024-03-29T05:28:54.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:28:54.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:28:54.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-29T05:28:54.984 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-29T05:28:54.984 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-29T05:28:55.042 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-03-29T05:28:55.042 DEBUG:teuthology.orchestra.run.smithi123:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-03-29T05:28:55.097 INFO:tasks.workunit.client.0.smithi123.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-29T05:28:55.102 INFO:tasks.workunit.client.0.smithi123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-29T05:28:55.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:28:55.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-03-29T05:28:55.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-03-29T05:28:55.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none ' 2024-03-29T05:28:55.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-03-29T05:28:55.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-03-29T05:28:55.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-29T05:28:55.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-03-29T05:28:55.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-29T05:28:55.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-03-29T05:28:55.107 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_recovery_scrub_2' 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.36690 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-29T05:28:55.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:28:55.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:28:55.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:28:55.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:28:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:28:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:28:55.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:28:55.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:28:55.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:28:55.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:28:55.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:28:55.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:28:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:28:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-29T05:28:55.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:28:55.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:55.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:28:55.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36690 2024-03-29T05:28:55.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:28:55.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:28:55.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-29T05:28:55.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:28:55.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:55.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:28:55.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36690 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:28:55.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:28:55.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-29T05:28:55.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-29T05:28:55.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:28:55.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:28:55.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:28:55.551 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:28:55.551 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:55.551 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:28:55.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:28:55.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-29T05:28:55.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:28:55.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:28:55.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:28:55.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:28:55.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:28:55.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:28:55.780 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:28:55.780 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:28:55.780 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:28:55.780 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:28:55.780 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:55.781 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:28:55.781 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36690/ceph-mon.a.asok 2024-03-29T05:28:55.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:28:55.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36690/ceph-mon.a.asok config get fsid 2024-03-29T05:28:55.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:28:55.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:28:55.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:28:55.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36690/ceph-mon.a.asok 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:28:55.894 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36690/ceph-mon.a.asok config get mon_host 2024-03-29T05:28:55.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-03-29T05:28:55.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-29T05:28:55.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:28:55.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:28:55.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:28:55.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-29T05:28:55.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:28:56.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:28:56.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:28:56.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:28:56.844 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:28:56.844 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:56.845 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:28:56.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:28:56.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:28:56.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:28:56.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-29T05:28:56.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-29T05:28:56.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-29T05:28:56.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-29T05:28:56.880 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-03-29T05:28:56.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-03-29T05:28:56.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:28:56.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-03-29T05:28:56.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:28:56.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:28:56.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:28:56.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:28:56.885 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:28:56.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-29T05:28:56.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-29T05:28:56.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:28:56.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e37e834-8a5e-4a16-a133-90b799d31b46 2024-03-29T05:28:56.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5e37e834-8a5e-4a16-a133-90b799d31b46' 2024-03-29T05:28:56.887 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 5e37e834-8a5e-4a16-a133-90b799d31b46 2024-03-29T05:28:56.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:28:56.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYUQZmoByuNRAAFtFK8p55imtckQaCDtq1Iw== 2024-03-29T05:28:56.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYUQZmoByuNRAAFtFK8p55imtckQaCDtq1Iw=="}' 2024-03-29T05:28:56.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e37e834-8a5e-4a16-a133-90b799d31b46 -i td/osd-recovery-scrub/0/new.json 2024-03-29T05:28:57.148 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:28:57.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-29T05:28:57.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCYUQZmoByuNRAAFtFK8p55imtckQaCDtq1Iw== --osd-uuid 5e37e834-8a5e-4a16-a133-90b799d31b46 2024-03-29T05:28:57.202 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:57.194+0000 7efd8b0547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:57.215 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:57.206+0000 7efd8b0547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:57.216 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:57.210+0000 7efd8b0547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:28:57.217 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:57.210+0000 7efd8b0547c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-29T05:28:57.217 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:57.210+0000 7efd8b0547c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-29T05:28:57.217 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:57.210+0000 7efd8b0547c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-29T05:28:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:28:57.322+0000 7efd8b0547c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-29T05:29:03.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-29T05:29:03.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:29:03.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:29:03.471 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:29:03.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:29:03.845 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:29:03.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:29:03.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:29:03.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-29T05:29:03.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:29:03.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:29:03.879 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:03.870+0000 7f4221a207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:03.905 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:03.898+0000 7f4221a207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:03.907 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:03.898+0000 7f4221a207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:29:04.165 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:29:04.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:04.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:29:04.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:04.469 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:04.462+0000 7f4221a207c0 -1 Falling back to public interface 2024-03-29T05:29:05.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:05.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:05.471 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:29:05.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:29:05.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:05.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:29:05.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:06.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:06.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:06.790 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:29:06.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:29:06.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:06.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:29:07.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:07.119 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:07.110+0000 7f4221a207c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:29:08.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:08.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:08.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:29:08.109 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:29:08.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:08.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:29:08.447 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1892937786,v1:127.0.0.1:6803/1892937786] [v2:127.0.0.1:6804/1892937786,v1:127.0.0.1:6805/1892937786] exists,up 5e37e834-8a5e-4a16-a133-90b799d31b46 2024-03-29T05:29:08.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:29:08.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:29:08.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:29:08.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:29:08.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-03-29T05:29:08.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:29:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:29:08.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:29:08.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:29:08.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:29:08.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:29:08.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-29T05:29:08.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-29T05:29:08.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:29:08.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86f7944b-007a-4531-83a1-e69844f42331 2024-03-29T05:29:08.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 86f7944b-007a-4531-83a1-e69844f42331' 2024-03-29T05:29:08.455 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 86f7944b-007a-4531-83a1-e69844f42331 2024-03-29T05:29:08.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:29:08.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkUQZmmYwYHBAAuLGvDfun7xsLOYyudtlOAw== 2024-03-29T05:29:08.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkUQZmmYwYHBAAuLGvDfun7xsLOYyudtlOAw=="}' 2024-03-29T05:29:08.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86f7944b-007a-4531-83a1-e69844f42331 -i td/osd-recovery-scrub/1/new.json 2024-03-29T05:29:08.840 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:29:08.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-29T05:29:08.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCkUQZmmYwYHBAAuLGvDfun7xsLOYyudtlOAw== --osd-uuid 86f7944b-007a-4531-83a1-e69844f42331 2024-03-29T05:29:08.897 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:08.890+0000 7f9d40b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:08.929 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:08.922+0000 7f9d40b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:08.931 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:08.922+0000 7f9d40b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:08.932 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:08.926+0000 7f9d40b497c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-29T05:29:08.932 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:08.926+0000 7f9d40b497c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-29T05:29:08.932 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:08.926+0000 7f9d40b497c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-29T05:29:08.999 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:08.994+0000 7f9d40b497c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-29T05:29:15.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-29T05:29:15.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:29:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:29:15.636 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:29:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:29:16.028 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:29:16.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:29:16.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:29:16.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:29:16.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-29T05:29:16.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:29:16.062 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:16.058+0000 7fe95fe797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:16.081 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:16.078+0000 7fe95fe797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:16.082 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:16.078+0000 7fe95fe797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:16.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:29:16.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:29:16.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:29:16.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:29:16.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:29:16.354 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:29:16.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:16.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:29:16.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:16.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:29:16.618 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:16.614+0000 7fe95fe797c0 -1 Falling back to public interface 2024-03-29T05:29:16.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:17.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:17.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:17.678 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:29:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:29:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:29:17.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:18.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:18.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:18.985 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:29:18.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:29:18.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:18.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:29:19.241 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:19.238+0000 7fe95fe797c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:29:19.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:20.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:20.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:20.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:29:20.372 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:29:20.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:20.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:29:20.691 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/659835192,v1:127.0.0.1:6811/659835192] [v2:127.0.0.1:6812/659835192,v1:127.0.0.1:6813/659835192] exists,up 86f7944b-007a-4531-83a1-e69844f42331 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-29T05:29:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:29:20.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:29:20.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-29T05:29:20.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:29:20.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba100ecf-f629-4ce0-a0c4-21eb626df7a7 2024-03-29T05:29:20.700 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 ba100ecf-f629-4ce0-a0c4-21eb626df7a7 2024-03-29T05:29:20.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ba100ecf-f629-4ce0-a0c4-21eb626df7a7' 2024-03-29T05:29:20.700 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:29:20.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwUQZmWJOpKhAA750hh3ftkrkJM7LzXTReVQ== 2024-03-29T05:29:20.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwUQZmWJOpKhAA750hh3ftkrkJM7LzXTReVQ=="}' 2024-03-29T05:29:20.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba100ecf-f629-4ce0-a0c4-21eb626df7a7 -i td/osd-recovery-scrub/2/new.json 2024-03-29T05:29:21.199 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:29:21.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-29T05:29:21.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCwUQZmWJOpKhAA750hh3ftkrkJM7LzXTReVQ== --osd-uuid ba100ecf-f629-4ce0-a0c4-21eb626df7a7 2024-03-29T05:29:21.256 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:21.254+0000 7fc25a5c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:21.301 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:21.298+0000 7fc25a5c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:21.303 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:21.298+0000 7fc25a5c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:21.303 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:21.298+0000 7fc25a5c17c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-29T05:29:21.303 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:21.298+0000 7fc25a5c17c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-29T05:29:21.303 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:21.298+0000 7fc25a5c17c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-29T05:29:21.377 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:21.374+0000 7fc25a5c17c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-29T05:29:28.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-29T05:29:28.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:29:28.362 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:29:28.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:29:28.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:29:28.749 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:29:28.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:29:28.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:29:28.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-29T05:29:28.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:29:28.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:29:28.782 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:28.778+0000 7f4fcc4be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:28.823 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:28.818+0000 7f4fcc4be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:28.825 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:28.822+0000 7f4fcc4be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:29.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:29:29.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:29:29.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:29:29.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:29:29.077 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:29:29.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:29:29.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:29.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:29:29.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:29.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:29:29.405 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:29.402+0000 7f4fcc4be7c0 -1 Falling back to public interface 2024-03-29T05:29:29.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:30.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:30.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:30.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:29:30.410 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:29:30.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:30.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:29:30.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:31.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:31.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:31.725 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:29:31.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:29:31.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:31.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:29:32.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:32.136 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:32.134+0000 7f4fcc4be7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:29:33.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:33.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:33.046 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:29:33.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:29:33.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:33.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:29:33.395 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2489670722,v1:127.0.0.1:6819/2489670722] [v2:127.0.0.1:6820/2489670722,v1:127.0.0.1:6821/2489670722] exists,up ba100ecf-f629-4ce0-a0c4-21eb626df7a7 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:29:33.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:29:33.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:29:33.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:29:33.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:29:33.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:29:33.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:29:33.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:29:33.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:29:33.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:29:33.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:29:33.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:29:33.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-29T05:29:33.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-29T05:29:33.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:29:33.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3b6a2277-ff19-4bb0-8903-f76d7acc3130 2024-03-29T05:29:33.407 INFO:tasks.workunit.client.0.smithi123.stdout:add osd3 3b6a2277-ff19-4bb0-8903-f76d7acc3130 2024-03-29T05:29:33.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3b6a2277-ff19-4bb0-8903-f76d7acc3130' 2024-03-29T05:29:33.407 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:29:33.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9UQZmj/ELGRAAxKd5BIx5zxKe7OljMUc+Eg== 2024-03-29T05:29:33.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9UQZmj/ELGRAAxKd5BIx5zxKe7OljMUc+Eg=="}' 2024-03-29T05:29:33.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3b6a2277-ff19-4bb0-8903-f76d7acc3130 -i td/osd-recovery-scrub/3/new.json 2024-03-29T05:29:33.809 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:29:33.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-29T05:29:33.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQC9UQZmj/ELGRAAxKd5BIx5zxKe7OljMUc+Eg== --osd-uuid 3b6a2277-ff19-4bb0-8903-f76d7acc3130 2024-03-29T05:29:33.862 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:33.858+0000 7fe2410327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:33.911 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:33.906+0000 7fe2410327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:33.912 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:33.910+0000 7fe2410327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:33.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:33.910+0000 7fe2410327c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-29T05:29:33.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:33.910+0000 7fe2410327c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-29T05:29:33.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:33.910+0000 7fe2410327c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-29T05:29:34.002 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:33.998+0000 7fe2410327c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-29T05:29:40.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-29T05:29:40.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:29:40.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-29T05:29:40.791 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd3 key to auth repository 2024-03-29T05:29:40.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:29:41.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-29T05:29:41.177 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.3 2024-03-29T05:29:41.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-29T05:29:41.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:29:41.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:29:41.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:29:41.209 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:41.206+0000 7f740550f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:41.211 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:41.206+0000 7f740550f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:41.212 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:41.210+0000 7f740550f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:29:41.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-29T05:29:41.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:29:41.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T05:29:41.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:29:41.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:29:41.511 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:29:41.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:41.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:29:41.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:29:41.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:41.750 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:41.746+0000 7f740550f7c0 -1 Falling back to public interface 2024-03-29T05:29:41.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:42.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:42.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:42.827 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:29:42.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:29:42.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:42.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:29:43.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:44.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:44.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:44.140 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:29:44.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:29:44.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:44.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:29:44.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:29:44.539 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:29:44.534+0000 7f740550f7c0 -1 osd.3 0 log_to_monitors true 2024-03-29T05:29:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:29:45.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:29:45.459 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:29:45.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:29:45.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:29:45.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:29:45.886 INFO:tasks.workunit.client.0.smithi123.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1492487064,v1:127.0.0.1:6827/1492487064] [v2:127.0.0.1:6828/1492487064,v1:127.0.0.1:6829/1492487064] exists,up 3b6a2277-ff19-4bb0-8903-f76d7acc3130 2024-03-29T05:29:45.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:29:45.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:29:45.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:29:45.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-03-29T05:29:45.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-03-29T05:29:46.433 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'test' created 2024-03-29T05:29:46.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:29:47.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-03-29T05:29:47.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:29:47.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:29:47.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:29:47.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:29:47.459 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:29:47.459 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:29:47.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:29:47.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:29:47.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:29:47.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:29:47.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:29:47.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:29:47.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:29:47.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:29:47.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:29:47.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:29:47.957 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:29:47.957 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:29:47.957 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T05:29:47.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:29:47.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:29:47.957 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:29:48.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-03-29T05:29:48.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-03-29T05:29:48.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-03-29T05:29:48.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:29:48.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:29:48.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-29T05:29:48.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-29T05:29:48.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-03-29T05:29:48.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:29:48.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:29:48.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-29T05:29:48.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-29T05:29:48.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444' 2024-03-29T05:29:48.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:29:48.389 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T05:29:48.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-03-29T05:29:48.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-03-29T05:29:48.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509444 3-85899345922' 2024-03-29T05:29:48.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:29:48.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-03-29T05:29:48.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:29:48.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:29:48.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-03-29T05:29:48.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:29:48.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-03-29T05:29:48.531 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836489 2024-03-29T05:29:48.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-03-29T05:29:48.532 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:29:48.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-03-29T05:29:48.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:29:49.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:29:49.830 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:29:50.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836489 2024-03-29T05:29:50.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:29:50.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:29:50.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-29T05:29:50.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:29:50.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-29T05:29:50.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:29:50.180 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672967 2024-03-29T05:29:50.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-29T05:29:50.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-29T05:29:50.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:29:50.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-03-29T05:29:50.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:29:50.498 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-29T05:29:50.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:29:50.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:29:50.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-29T05:29:50.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:29:50.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-29T05:29:50.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-29T05:29:50.503 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509444 2024-03-29T05:29:50.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:29:50.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-03-29T05:29:50.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:29:50.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-03-29T05:29:50.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:29:50.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T05:29:50.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-03-29T05:29:50.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:29:50.856 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345922 2024-03-29T05:29:50.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-03-29T05:29:50.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-03-29T05:29:50.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T05:29:51.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-03-29T05:29:51.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:29:51.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:29:51.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:29:51.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:29:51.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:29:51.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:29:51.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:29:51.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:29:51.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:29:51.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:29:51.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:29:52.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:29:52.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:29:52.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:29:52.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:29:52.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:29:52.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:29:52.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:29:52.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-03-29T05:29:52.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-03-29T05:29:52.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-03-29T05:29:52.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-03-29T05:29:52.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-29T05:29:53.089 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:29:53.090 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:29:47.749454+0000 0'0 24:12 [1] 1 [1] 1 0'0 2024-03-29T05:29:46.272550+0000 0'0 2024-03-29T05:29:46.272550+0000 0 0 periodic scrub scheduled @ 2024-03-30T07:45:55.256468+0000 0 0 2024-03-29T05:29:53.090 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:29:53.090 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:29:53.090 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:29:53.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.36690 bs=1M count=50 2024-03-29T05:29:53.267 INFO:tasks.workunit.client.0.smithi123.stderr:50+0 records in 2024-03-29T05:29:53.267 INFO:tasks.workunit.client.0.smithi123.stderr:50+0 records out 2024-03-29T05:29:53.267 INFO:tasks.workunit.client.0.smithi123.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.159216 s, 329 MB/s 2024-03-29T05:29:53.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-03-29T05:29:53.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:29:53.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.36690 2024-03-29T05:29:55.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:29:55.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.36690 2024-03-29T05:29:57.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:29:57.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.36690 2024-03-29T05:29:58.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:29:58.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.36690 2024-03-29T05:30:00.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:00.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.36690 2024-03-29T05:30:02.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:02.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.36690 2024-03-29T05:30:04.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:04.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.36690 2024-03-29T05:30:06.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:06.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.36690 2024-03-29T05:30:08.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:08.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.36690 2024-03-29T05:30:09.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:09.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.36690 2024-03-29T05:30:11.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:11.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.36690 2024-03-29T05:30:13.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:13.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.36690 2024-03-29T05:30:15.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:15.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.36690 2024-03-29T05:30:17.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:17.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.36690 2024-03-29T05:30:19.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:19.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.36690 2024-03-29T05:30:21.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:21.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.36690 2024-03-29T05:30:22.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:22.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.36690 2024-03-29T05:30:24.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:24.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.36690 2024-03-29T05:30:26.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:26.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.36690 2024-03-29T05:30:28.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:28.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.36690 2024-03-29T05:30:30.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:30.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.36690 2024-03-29T05:30:32.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:32.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.36690 2024-03-29T05:30:34.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:34.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.36690 2024-03-29T05:30:36.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:36.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.36690 2024-03-29T05:30:37.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:37.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.36690 2024-03-29T05:30:39.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:39.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.36690 2024-03-29T05:30:41.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:41.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.36690 2024-03-29T05:30:43.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:43.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.36690 2024-03-29T05:30:45.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:45.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.36690 2024-03-29T05:30:47.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:47.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.36690 2024-03-29T05:30:49.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:49.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.36690 2024-03-29T05:30:50.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:50.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.36690 2024-03-29T05:30:52.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:52.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.36690 2024-03-29T05:30:54.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:54.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.36690 2024-03-29T05:30:56.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:56.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.36690 2024-03-29T05:30:58.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:58.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.36690 2024-03-29T05:30:59.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:30:59.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.36690 2024-03-29T05:31:01.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:01.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.36690 2024-03-29T05:31:03.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:03.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.36690 2024-03-29T05:31:05.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:05.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.36690 2024-03-29T05:31:07.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:07.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.36690 2024-03-29T05:31:09.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:09.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.36690 2024-03-29T05:31:11.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:11.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.36690 2024-03-29T05:31:12.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:12.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.36690 2024-03-29T05:31:14.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:14.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.36690 2024-03-29T05:31:16.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:16.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.36690 2024-03-29T05:31:18.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:18.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.36690 2024-03-29T05:31:20.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:20.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.36690 2024-03-29T05:31:22.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:22.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.36690 2024-03-29T05:31:24.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:24.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.36690 2024-03-29T05:31:25.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:25.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.36690 2024-03-29T05:31:27.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:27.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.36690 2024-03-29T05:31:29.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:29.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.36690 2024-03-29T05:31:31.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:31.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.36690 2024-03-29T05:31:33.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:33.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.36690 2024-03-29T05:31:35.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:35.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.36690 2024-03-29T05:31:36.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:36.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.36690 2024-03-29T05:31:38.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:38.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.36690 2024-03-29T05:31:40.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:40.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.36690 2024-03-29T05:31:42.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:42.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.36690 2024-03-29T05:31:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.36690 2024-03-29T05:31:46.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:46.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.36690 2024-03-29T05:31:48.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:48.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.36690 2024-03-29T05:31:50.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:50.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.36690 2024-03-29T05:31:51.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:51.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.36690 2024-03-29T05:31:53.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:53.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.36690 2024-03-29T05:31:55.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:55.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.36690 2024-03-29T05:31:57.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:57.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.36690 2024-03-29T05:31:59.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:31:59.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.36690 2024-03-29T05:32:01.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:01.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.36690 2024-03-29T05:32:03.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:03.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.36690 2024-03-29T05:32:04.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:04.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.36690 2024-03-29T05:32:06.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:06.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.36690 2024-03-29T05:32:08.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:08.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.36690 2024-03-29T05:32:10.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:10.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.36690 2024-03-29T05:32:12.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:12.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.36690 2024-03-29T05:32:14.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:14.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.36690 2024-03-29T05:32:16.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:16.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.36690 2024-03-29T05:32:17.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:17.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.36690 2024-03-29T05:32:19.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:19.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.36690 2024-03-29T05:32:21.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:21.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.36690 2024-03-29T05:32:23.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:23.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.36690 2024-03-29T05:32:24.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:24.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.36690 2024-03-29T05:32:26.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:26.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.36690 2024-03-29T05:32:28.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:28.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.36690 2024-03-29T05:32:30.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:30.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.36690 2024-03-29T05:32:32.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:32.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.36690 2024-03-29T05:32:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.36690 2024-03-29T05:32:36.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:36.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.36690 2024-03-29T05:32:37.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:37.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.36690 2024-03-29T05:32:39.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:39.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.36690 2024-03-29T05:32:41.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:41.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.36690 2024-03-29T05:32:43.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:43.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.36690 2024-03-29T05:32:45.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:45.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.36690 2024-03-29T05:32:46.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:46.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.36690 2024-03-29T05:32:48.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:48.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.36690 2024-03-29T05:32:50.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:50.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.36690 2024-03-29T05:32:52.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:52.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.36690 2024-03-29T05:32:54.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:54.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.36690 2024-03-29T05:32:55.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-29T05:32:55.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.36690 2024-03-29T05:32:57.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.36690 2024-03-29T05:32:57.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-03-29T05:32:58.283 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 4 2024-03-29T05:32:58.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-03-29T05:32:58.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-03-29T05:32:58.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-29T05:32:58.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-29T05:32:58.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-29T05:32:58.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-29T05:32:58.582 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:32:58.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-29T05:33:00.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-03-29T05:33:00.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-03-29T05:33:00.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-03-29T05:33:00.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-29T05:33:00.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-29T05:33:00.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-29T05:33:00.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-29T05:33:00.916 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:33:00.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-29T05:33:02.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-03-29T05:33:02.935 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-03-29T05:33:02.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-03-29T05:33:02.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-29T05:33:02.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-29T05:33:02.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-29T05:33:02.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-29T05:33:03.294 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:33:03.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-03-29T05:33:03.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-03-29T05:33:03.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-29T05:33:03.605 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:33:03.606 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-03-29T05:33:00.591122+0000 24'1300 27:1324 [1,0,2,3] 1 [1,0] 1 0'0 2024-03-29T05:29:46.272550+0000 0'0 2024-03-29T05:29:46.272550+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:33:03.606 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:33:03.606 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:33:03.606 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:33:03.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-03-29T05:33:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-03-29T05:33:13.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:33:13.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:33:13.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:33:13.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:33:13.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:33:14.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:33:14.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-03-29T05:33:14.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-03-29T05:33:14.943 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-03-29T05:33:14.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-03-29T05:33:14.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:14.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-29T05:33:14.949 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:09.020+0000 7f421a9d4640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.949 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:11.976+0000 7f421a9d4640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:14.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-29T05:33:14.964 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:02.148+0000 7fe958e2d640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.964 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:04.120+0000 7fe958e2d640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.965 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:06.160+0000 7fe958e2d640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.965 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:13.056+0000 7fe958e2d640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:14.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-29T05:33:14.969 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:06.232+0000 7f4fc5472640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.969 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:08.212+0000 7f4fc5472640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.969 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:10.236+0000 7f4fc5472640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:14.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-29T05:33:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:01.520+0000 7f73fe4c3640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:02.516+0000 7f73fe4c3640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:07.516+0000 7f73fe4c3640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:08.512+0000 7f73fe4c3640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:09.544+0000 7f73fe4c3640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:11.548+0000 7f73fe4c3640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:33:13.476+0000 7f73fe4c3640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-29T05:33:14.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-03-29T05:33:14.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-03-29T05:33:14.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-03-29T05:33:14.973 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-03-29T05:33:14.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-03-29T05:33:14.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:14.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-29T05:33:14.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-29T05:33:14.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-03-29T05:33:14.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-03-29T05:33:14.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:14.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-29T05:33:14.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-29T05:33:14.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-03-29T05:33:14.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-03-29T05:33:14.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:14.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-29T05:33:15.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-29T05:33:15.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-03-29T05:33:15.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-03-29T05:33:15.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:15.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-29T05:33:15.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-29T05:33:15.005 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-03-29T05:33:15.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-03-29T05:33:15.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:33:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:33:15.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:33:15.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:33:15.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:33:15.016 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:33:15.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:33:15.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:33:15.017 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:33:15.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:33:15.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:33:15.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:33:15.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:33:15.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:33:15.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:33:15.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-29T05:33:15.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:33:15.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:15.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:15.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36690 2024-03-29T05:33:15.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:33:15.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:33:15.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-03-29T05:33:15.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-03-29T05:33:15.046 INFO:tasks.workunit.client.0.smithi123.stdout:TEST PASSED 2024-03-29T05:33:15.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.36690 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:33:15.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-29T05:33:15.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:33:15.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:33:15.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:33:15.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:33:15.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:33:15.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:33:15.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:33:15.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:33:15.050 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:33:15.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:33:15.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:33:15.050 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:33:15.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:33:15.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:33:15.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:33:15.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:33:15.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:33:15.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:33:15.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-29T05:33:15.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:33:15.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:15.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36690 2024-03-29T05:33:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:33:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:33:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-29T05:33:15.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:33:15.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:15.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.36690 2024-03-29T05:33:15.058 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:33:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:33:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-29T05:33:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-29T05:33:15.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:33:15.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:33:15.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:33:15.445 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:33:15.445 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:15.445 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:15.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:15.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-29T05:33:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:33:15.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:33:15.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:33:15.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:33:15.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:33:15.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:33:15.639 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:33:15.639 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:33:15.639 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:33:15.639 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:33:15.639 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:15.640 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:15.640 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36690/ceph-mon.a.asok 2024-03-29T05:33:15.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:33:15.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36690/ceph-mon.a.asok config get fsid 2024-03-29T05:33:15.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:33:15.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:33:15.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:33:15.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:33:15.757 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:33:15.757 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:33:15.757 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:33:15.757 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:33:15.757 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:33:15.757 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:15.757 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:15.758 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.36690/ceph-mon.a.asok 2024-03-29T05:33:15.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:33:15.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.36690/ceph-mon.a.asok config get mon_host 2024-03-29T05:33:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-03-29T05:33:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-29T05:33:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:33:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:33:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:33:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-29T05:33:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:33:16.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:33:16.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:33:16.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:33:16.747 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:33:16.747 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:16.747 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:16.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:16.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:33:16.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:33:16.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-29T05:33:16.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-29T05:33:16.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-29T05:33:16.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-29T05:33:16.783 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-03-29T05:33:16.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-03-29T05:33:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:33:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:33:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:33:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:33:16.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:33:16.787 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:33:16.787 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:16.787 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:16.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:33:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:33:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:33:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:33:16.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9e449e2-6313-4fab-b79c-cc5af99aabca 2024-03-29T05:33:16.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e9e449e2-6313-4fab-b79c-cc5af99aabca' 2024-03-29T05:33:16.790 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 e9e449e2-6313-4fab-b79c-cc5af99aabca 2024-03-29T05:33:16.790 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:33:16.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcUgZmqrEUMBAAQLD1QszDeIbH17iw5AbZwA== 2024-03-29T05:33:16.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcUgZmqrEUMBAAQLD1QszDeIbH17iw5AbZwA=="}' 2024-03-29T05:33:16.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9e449e2-6313-4fab-b79c-cc5af99aabca -i td/osd-recovery-scrub/0/new.json 2024-03-29T05:33:17.075 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:33:17.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-29T05:33:17.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCcUgZmqrEUMBAAQLD1QszDeIbH17iw5AbZwA== --osd-uuid e9e449e2-6313-4fab-b79c-cc5af99aabca 2024-03-29T05:33:17.128 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:17.124+0000 7f4e0a17e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:17.142 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:17.140+0000 7f4e0a17e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:17.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:17.140+0000 7f4e0a17e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:17.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:17.140+0000 7f4e0a17e7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-29T05:33:17.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:17.140+0000 7f4e0a17e7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-29T05:33:17.144 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:17.140+0000 7f4e0a17e7c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-29T05:33:17.234 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:17.232+0000 7f4e0a17e7c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-29T05:33:23.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-29T05:33:23.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:33:23.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:33:23.519 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:33:23.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:33:23.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:33:23.887 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:33:23.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:33:23.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:33:23.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:33:23.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:33:23.917 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:23.916+0000 7f19f315d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:23.919 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:23.916+0000 7f19f315d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:23.920 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:23.920+0000 7f19f315d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:24.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:33:24.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:33:24.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:33:24.175 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:33:24.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:33:24.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:33:24.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:24.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:33:24.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:24.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:33:24.463 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:24.460+0000 7f19f315d7c0 -1 Falling back to public interface 2024-03-29T05:33:24.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:25.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:25.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:25.527 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:33:25.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:33:25.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:25.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:33:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:26.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:26.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:26.847 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:33:26.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:33:26.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:26.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:33:27.131 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:27.128+0000 7f19f315d7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:33:27.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:28.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:28.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:28.164 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:33:28.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:33:28.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:28.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:33:28.475 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3285611112,v1:127.0.0.1:6803/3285611112] [v2:127.0.0.1:6804/3285611112,v1:127.0.0.1:6805/3285611112] exists,up e9e449e2-6313-4fab-b79c-cc5af99aabca 2024-03-29T05:33:28.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:33:28.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:33:28.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:33:28.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:28.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:33:28.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:33:28.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:33:28.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:33:28.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:33:28.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:33:28.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-29T05:33:28.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:33:28.481 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 2adc298a-be20-40af-b2ff-5b4b91ef0fd7 2024-03-29T05:33:28.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2adc298a-be20-40af-b2ff-5b4b91ef0fd7 2024-03-29T05:33:28.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2adc298a-be20-40af-b2ff-5b4b91ef0fd7' 2024-03-29T05:33:28.482 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:33:28.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoUgZmS1yjHRAAukpN3xvPI/xRoZHDzF4UNA== 2024-03-29T05:33:28.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoUgZmS1yjHRAAukpN3xvPI/xRoZHDzF4UNA=="}' 2024-03-29T05:33:28.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2adc298a-be20-40af-b2ff-5b4b91ef0fd7 -i td/osd-recovery-scrub/1/new.json 2024-03-29T05:33:28.849 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:33:28.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-29T05:33:28.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCoUgZmS1yjHRAAukpN3xvPI/xRoZHDzF4UNA== --osd-uuid 2adc298a-be20-40af-b2ff-5b4b91ef0fd7 2024-03-29T05:33:28.906 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:28.904+0000 7f025fdef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:28.950 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:28.948+0000 7f025fdef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:28.952 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:28.948+0000 7f025fdef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:28.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:28.952+0000 7f025fdef7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-29T05:33:28.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:28.952+0000 7f025fdef7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-29T05:33:28.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:28.952+0000 7f025fdef7c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-29T05:33:29.022 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:29.024+0000 7f025fdef7c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-29T05:33:35.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-29T05:33:35.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:33:35.783 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:33:35.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:33:35.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:33:36.176 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:33:36.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:33:36.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:33:36.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:33:36.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:33:36.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:33:36.212 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:36.212+0000 7fbdfb4427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:36.249 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:36.248+0000 7fbdfb4427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:36.251 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:36.248+0000 7fbdfb4427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:36.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:33:36.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:33:36.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:33:36.530 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:33:36.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:33:36.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:33:36.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:36.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:33:36.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:33:36.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:36.811 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:36.808+0000 7fbdfb4427c0 -1 Falling back to public interface 2024-03-29T05:33:36.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:37.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:37.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:37.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:33:37.879 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:33:37.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:37.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:33:38.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:39.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:39.190 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:33:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:33:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:33:39.493 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:39.492+0000 7fbdfb4427c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:33:39.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:40.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:40.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:40.506 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:33:40.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:33:40.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:40.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:33:40.905 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1462906250,v1:127.0.0.1:6811/1462906250] [v2:127.0.0.1:6812/1462906250,v1:127.0.0.1:6813/1462906250] exists,up 2adc298a-be20-40af-b2ff-5b4b91ef0fd7 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:33:40.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:40.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:33:40.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-29T05:33:40.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:33:40.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31179254-1161-4fb9-a8ee-88a09d2f76f5 2024-03-29T05:33:40.912 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 31179254-1161-4fb9-a8ee-88a09d2f76f5 2024-03-29T05:33:40.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 31179254-1161-4fb9-a8ee-88a09d2f76f5' 2024-03-29T05:33:40.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:33:40.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0UgZmJuKpNxAA4VJV5xNxVA6AadFjYstydg== 2024-03-29T05:33:40.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0UgZmJuKpNxAA4VJV5xNxVA6AadFjYstydg=="}' 2024-03-29T05:33:40.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31179254-1161-4fb9-a8ee-88a09d2f76f5 -i td/osd-recovery-scrub/2/new.json 2024-03-29T05:33:41.311 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:33:41.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-29T05:33:41.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC0UgZmJuKpNxAA4VJV5xNxVA6AadFjYstydg== --osd-uuid 31179254-1161-4fb9-a8ee-88a09d2f76f5 2024-03-29T05:33:41.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:41.372+0000 7f012d12a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:41.410 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:41.408+0000 7f012d12a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:41.412 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:41.408+0000 7f012d12a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:41.412 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:41.412+0000 7f012d12a7c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-29T05:33:41.413 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:41.412+0000 7f012d12a7c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-29T05:33:41.413 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:41.412+0000 7f012d12a7c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-29T05:33:41.492 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:41.488+0000 7f012d12a7c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-29T05:33:48.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-29T05:33:48.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:33:48.550 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:33:48.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:33:48.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:33:48.939 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:33:48.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:33:48.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:33:48.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:33:48.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:33:48.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:33:48.975 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:48.971+0000 7f8aee2177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:49.002 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:48.999+0000 7f8aee2177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:49.003 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:48.999+0000 7f8aee2177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:49.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:33:49.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:33:49.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:33:49.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:33:49.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:33:49.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:49.329 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:33:49.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:33:49.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:49.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:33:49.559 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:49.555+0000 7f8aee2177c0 -1 Falling back to public interface 2024-03-29T05:33:49.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:50.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:50.674 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:33:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:33:50.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:50.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:33:50.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:51.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:51.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:51.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:33:51.986 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:33:51.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:51.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:33:52.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:52.283+0000 7f8aee2177c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:33:52.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:33:53.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:33:53.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:33:53.304 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:33:53.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:33:53.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:33:53.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:33:53.662 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2461083301,v1:127.0.0.1:6819/2461083301] [v2:127.0.0.1:6820/2461083301,v1:127.0.0.1:6821/2461083301] exists,up 31179254-1161-4fb9-a8ee-88a09d2f76f5 2024-03-29T05:33:53.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:33:53.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:33:53.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:33:53.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:33:53.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:33:53.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:33:53.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-29T05:33:53.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:33:53.667 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:33:53.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:53.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:33:53.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:33:53.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:33:53.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-29T05:33:53.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:33:53.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88ec904a-1044-451b-abbe-8f3d291ee7d1 2024-03-29T05:33:53.671 INFO:tasks.workunit.client.0.smithi123.stdout:add osd3 88ec904a-1044-451b-abbe-8f3d291ee7d1 2024-03-29T05:33:53.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 88ec904a-1044-451b-abbe-8f3d291ee7d1' 2024-03-29T05:33:53.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:33:53.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBUgZmgbI6KRAARxX5d8hErrxL50gjihq7KA== 2024-03-29T05:33:53.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBUgZmgbI6KRAARxX5d8hErrxL50gjihq7KA=="}' 2024-03-29T05:33:53.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88ec904a-1044-451b-abbe-8f3d291ee7d1 -i td/osd-recovery-scrub/3/new.json 2024-03-29T05:33:54.072 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:33:54.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-29T05:33:54.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDBUgZmgbI6KRAARxX5d8hErrxL50gjihq7KA== --osd-uuid 88ec904a-1044-451b-abbe-8f3d291ee7d1 2024-03-29T05:33:54.130 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:54.127+0000 7f71353a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:54.173 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:54.171+0000 7f71353a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:54.175 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:54.171+0000 7f71353a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:33:54.175 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:54.171+0000 7f71353a77c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-29T05:33:54.176 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:54.171+0000 7f71353a77c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-29T05:33:54.176 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:54.171+0000 7f71353a77c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-29T05:33:54.272 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:33:54.267+0000 7f71353a77c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-29T05:34:01.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-29T05:34:01.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:34:01.341 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd3 key to auth repository 2024-03-29T05:34:01.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-29T05:34:01.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:34:01.733 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.3 2024-03-29T05:34:01.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-29T05:34:01.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:34:01.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:01.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:34:01.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:34:01.771 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:01.767+0000 7f7083f847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:01.798 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:01.795+0000 7f7083f847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:01.799 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:01.795+0000 7f7083f847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:02.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-29T05:34:02.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:34:02.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T05:34:02.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:34:02.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:34:02.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:02.044 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:34:02.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:34:02.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:02.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:34:02.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:02.366 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:02.363+0000 7f7083f847c0 -1 Falling back to public interface 2024-03-29T05:34:03.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:03.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:03.341 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:34:03.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:34:03.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:03.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:34:03.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:04.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:04.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:04.666 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:34:04.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:34:04.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:04.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:34:04.959 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:04.955+0000 7f7083f847c0 -1 osd.3 0 log_to_monitors true 2024-03-29T05:34:04.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:05.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:05.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:05.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:34:05.982 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:34:05.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:05.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T05:34:06.313 INFO:tasks.workunit.client.0.smithi123.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1986980855,v1:127.0.0.1:6827/1986980855] [v2:127.0.0.1:6828/1986980855,v1:127.0.0.1:6829/1986980855] exists,up 88ec904a-1044-451b-abbe-8f3d291ee7d1 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-29T05:34:06.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:34:06.317 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:34:06.318 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:34:06.318 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:34:06.318 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:34:06.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-03-29T05:34:06.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:34:06.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=611a877a-03b3-43fb-8aba-58e50382b5d2 2024-03-29T05:34:06.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 611a877a-03b3-43fb-8aba-58e50382b5d2' 2024-03-29T05:34:06.321 INFO:tasks.workunit.client.0.smithi123.stdout:add osd4 611a877a-03b3-43fb-8aba-58e50382b5d2 2024-03-29T05:34:06.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:34:06.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOUgZmAdUyFBAA7bVm1dyZTOuySbbQgngx/w== 2024-03-29T05:34:06.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOUgZmAdUyFBAA7bVm1dyZTOuySbbQgngx/w=="}' 2024-03-29T05:34:06.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 611a877a-03b3-43fb-8aba-58e50382b5d2 -i td/osd-recovery-scrub/4/new.json 2024-03-29T05:34:06.728 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:34:06.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-03-29T05:34:06.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDOUgZmAdUyFBAA7bVm1dyZTOuySbbQgngx/w== --osd-uuid 611a877a-03b3-43fb-8aba-58e50382b5d2 2024-03-29T05:34:06.792 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:06.787+0000 7fd8125407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:06.795 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:06.791+0000 7fd8125407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:06.797 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:06.795+0000 7fd8125407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:06.797 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:06.795+0000 7fd8125407c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-29T05:34:06.797 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:06.795+0000 7fd8125407c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-29T05:34:06.797 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:06.795+0000 7fd8125407c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-29T05:34:06.896 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:06.891+0000 7fd8125407c0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-03-29T05:34:13.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-03-29T05:34:13.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:34:13.805 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd4 key to auth repository 2024-03-29T05:34:13.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-29T05:34:13.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:34:14.198 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.4 2024-03-29T05:34:14.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-29T05:34:14.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:14.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:34:14.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:34:14.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:34:14.233 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:14.231+0000 7f7f15eae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:14.272 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:14.267+0000 7f7f15eae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:14.274 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:14.271+0000 7f7f15eae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:14.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-29T05:34:14.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:34:14.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-29T05:34:14.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:34:14.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:34:14.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:14.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:34:14.549 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:34:14.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:14.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T05:34:14.830 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:14.827+0000 7f7f15eae7c0 -1 Falling back to public interface 2024-03-29T05:34:14.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:34:15.872 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:34:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T05:34:16.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:17.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:17.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:17.180 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:34:17.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:34:17.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:17.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T05:34:17.500 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:17.495+0000 7f7f15eae7c0 -1 osd.4 0 log_to_monitors true 2024-03-29T05:34:17.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:18.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:18.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:18.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:34:18.521 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:34:18.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:18.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T05:34:18.842 INFO:tasks.workunit.client.0.smithi123.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/424706937,v1:127.0.0.1:6835/424706937] [v2:127.0.0.1:6836/424706937,v1:127.0.0.1:6837/424706937] exists,up 611a877a-03b3-43fb-8aba-58e50382b5d2 2024-03-29T05:34:18.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:34:18.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:34:18.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:34:18.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:34:18.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:34:18.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:34:18.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:18.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:18.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-03-29T05:34:18.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:34:18.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2559c72e-e437-415f-8efd-612ab9028e16 2024-03-29T05:34:18.850 INFO:tasks.workunit.client.0.smithi123.stdout:add osd5 2559c72e-e437-415f-8efd-612ab9028e16 2024-03-29T05:34:18.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 2559c72e-e437-415f-8efd-612ab9028e16' 2024-03-29T05:34:18.851 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:34:18.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaUgZm09AONBAA6VLcwdHdyoRkCXzVzq1Tzg== 2024-03-29T05:34:18.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaUgZm09AONBAA6VLcwdHdyoRkCXzVzq1Tzg=="}' 2024-03-29T05:34:18.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2559c72e-e437-415f-8efd-612ab9028e16 -i td/osd-recovery-scrub/5/new.json 2024-03-29T05:34:19.242 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:34:19.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-03-29T05:34:19.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDaUgZm09AONBAA6VLcwdHdyoRkCXzVzq1Tzg== --osd-uuid 2559c72e-e437-415f-8efd-612ab9028e16 2024-03-29T05:34:19.302 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:19.299+0000 7ff0f3ff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:19.309 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:19.307+0000 7ff0f3ff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:19.311 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:19.307+0000 7ff0f3ff77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:19.311 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:19.307+0000 7ff0f3ff77c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-29T05:34:19.311 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:19.307+0000 7ff0f3ff77c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-29T05:34:19.311 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:19.307+0000 7ff0f3ff77c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-29T05:34:19.443 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:19.439+0000 7ff0f3ff77c0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-03-29T05:34:26.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-03-29T05:34:26.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:34:26.411 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd5 key to auth repository 2024-03-29T05:34:26.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-29T05:34:26.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:34:26.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-29T05:34:26.792 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.5 2024-03-29T05:34:26.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:26.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:34:26.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:34:26.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:34:26.828 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:26.823+0000 7f41bebf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:26.853 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:26.851+0000 7f41bebf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:26.854 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:26.851+0000 7f41bebf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:27.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-29T05:34:27.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:34:27.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T05:34:27.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:34:27.132 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:34:27.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:34:27.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:27.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:34:27.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:27.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T05:34:27.403 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:27.399+0000 7f41bebf77c0 -1 Falling back to public interface 2024-03-29T05:34:27.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:28.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:28.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:28.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:34:28.456 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:34:28.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:28.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T05:34:28.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:29.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:29.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:29.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:34:29.764 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:34:29.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:29.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T05:34:30.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:30.177 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:30.175+0000 7f41bebf77c0 -1 osd.5 0 log_to_monitors true 2024-03-29T05:34:31.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:31.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:31.114 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:34:31.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:34:31.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:31.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T05:34:31.435 INFO:tasks.workunit.client.0.smithi123.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/544504368,v1:127.0.0.1:6843/544504368] [v2:127.0.0.1:6844/544504368,v1:127.0.0.1:6845/544504368] exists,up 2559c72e-e437-415f-8efd-612ab9028e16 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:34:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:31.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:34:31.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-03-29T05:34:31.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:34:31.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bee1dd06-26c3-4ac7-aec2-aa59ddd75c4f 2024-03-29T05:34:31.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 bee1dd06-26c3-4ac7-aec2-aa59ddd75c4f' 2024-03-29T05:34:31.442 INFO:tasks.workunit.client.0.smithi123.stdout:add osd6 bee1dd06-26c3-4ac7-aec2-aa59ddd75c4f 2024-03-29T05:34:31.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:34:31.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnUgZmsO2fGxAA5ZLVUvF1b0pQ/JecfoICog== 2024-03-29T05:34:31.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnUgZmsO2fGxAA5ZLVUvF1b0pQ/JecfoICog=="}' 2024-03-29T05:34:31.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bee1dd06-26c3-4ac7-aec2-aa59ddd75c4f -i td/osd-recovery-scrub/6/new.json 2024-03-29T05:34:31.893 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:34:31.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-03-29T05:34:31.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDnUgZmsO2fGxAA5ZLVUvF1b0pQ/JecfoICog== --osd-uuid bee1dd06-26c3-4ac7-aec2-aa59ddd75c4f 2024-03-29T05:34:31.954 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:31.950+0000 7fc914def7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:31.986 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:31.982+0000 7fc914def7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:31.987 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:31.982+0000 7fc914def7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:31.988 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:31.986+0000 7fc914def7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-29T05:34:31.988 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:31.986+0000 7fc914def7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-29T05:34:31.988 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:31.986+0000 7fc914def7c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-29T05:34:32.103 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:32.102+0000 7fc914def7c0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-03-29T05:34:39.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-03-29T05:34:39.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:34:39.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-29T05:34:39.050 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd6 key to auth repository 2024-03-29T05:34:39.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:34:39.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-29T05:34:39.432 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.6 2024-03-29T05:34:39.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:39.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:34:39.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:34:39.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:34:39.469 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:39.466+0000 7fa387e237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:39.479 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:39.474+0000 7fa387e237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:39.481 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:39.478+0000 7fa387e237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:39.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-29T05:34:39.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:34:39.745 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:34:39.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-29T05:34:39.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:34:39.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:34:39.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:39.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:34:39.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:39.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T05:34:40.050 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:40.046+0000 7fa387e237c0 -1 Falling back to public interface 2024-03-29T05:34:40.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:41.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:41.076 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:34:41.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:41.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:34:41.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:41.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T05:34:41.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:42.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:42.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:42.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:34:42.383 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:34:42.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:42.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T05:34:42.670 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:42.666+0000 7fa387e237c0 -1 osd.6 0 log_to_monitors true 2024-03-29T05:34:42.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:43.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:43.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:43.696 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:34:43.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:34:43.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:43.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T05:34:44.024 INFO:tasks.workunit.client.0.smithi123.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/295745382,v1:127.0.0.1:6851/295745382] [v2:127.0.0.1:6852/295745382,v1:127.0.0.1:6853/295745382] exists,up bee1dd06-26c3-4ac7-aec2-aa59ddd75c4f 2024-03-29T05:34:44.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:34:44.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' 2024-03-29T05:34:44.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-29T05:34:44.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-03-29T05:34:44.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:34:44.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=abba1c5f-d6c4-4340-8c43-de50fad5258c 2024-03-29T05:34:44.031 INFO:tasks.workunit.client.0.smithi123.stdout:add osd7 abba1c5f-d6c4-4340-8c43-de50fad5258c 2024-03-29T05:34:44.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 abba1c5f-d6c4-4340-8c43-de50fad5258c' 2024-03-29T05:34:44.032 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:34:44.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0UgZmyVUSAxAAMB7Pu6vMiRfReCZO34ZEQg== 2024-03-29T05:34:44.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0UgZmyVUSAxAAMB7Pu6vMiRfReCZO34ZEQg=="}' 2024-03-29T05:34:44.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new abba1c5f-d6c4-4340-8c43-de50fad5258c -i td/osd-recovery-scrub/7/new.json 2024-03-29T05:34:44.428 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:34:44.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-03-29T05:34:44.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQD0UgZmyVUSAxAAMB7Pu6vMiRfReCZO34ZEQg== --osd-uuid abba1c5f-d6c4-4340-8c43-de50fad5258c 2024-03-29T05:34:44.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:44.482+0000 7f96c8f2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:44.530 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:44.526+0000 7f96c8f2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:44.531 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:44.526+0000 7f96c8f2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:44.532 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:44.530+0000 7f96c8f2e7c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-29T05:34:44.532 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:44.530+0000 7f96c8f2e7c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-29T05:34:44.532 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:44.530+0000 7f96c8f2e7c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-29T05:34:44.629 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:44.630+0000 7f96c8f2e7c0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-03-29T05:34:51.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-03-29T05:34:51.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:34:51.834 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd7 key to auth repository 2024-03-29T05:34:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-29T05:34:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:34:52.226 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.7 2024-03-29T05:34:52.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-29T05:34:52.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:34:52.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=c69667d8-3ee4-405f-973a-827349fa7e28 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.36690/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-29T05:34:52.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:34:52.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:34:52.263 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:52.262+0000 7fe1c95567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:52.293 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:52.294+0000 7fe1c95567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:52.294 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:52.294+0000 7fe1c95567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:34:52.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-29T05:34:52.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:34:52.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-29T05:34:52.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:34:52.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:34:52.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:52.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:34:52.552 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:34:52.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T05:34:52.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:52.851 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:52.850+0000 7fe1c95567c0 -1 Falling back to public interface 2024-03-29T05:34:52.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:53.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:53.880 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:34:53.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:53.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:34:53.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:53.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T05:34:54.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:55.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:55.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:55.192 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:34:55.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:34:55.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:55.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T05:34:55.474 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:34:55.474+0000 7fe1c95567c0 -1 osd.7 0 log_to_monitors true 2024-03-29T05:34:55.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:34:56.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:34:56.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:34:56.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:34:56.570 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:34:56.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:34:56.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T05:34:56.910 INFO:tasks.workunit.client.0.smithi123.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/444620729,v1:127.0.0.1:6859/444620729] [v2:127.0.0.1:6860/444620729,v1:127.0.0.1:6861/444620729] exists,up abba1c5f-d6c4-4340-8c43-de50fad5258c 2024-03-29T05:34:56.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:34:56.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:34:56.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:34:56.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-03-29T05:34:56.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-03-29T05:34:57.319 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'test' created 2024-03-29T05:34:57.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:34:58.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-03-29T05:34:58.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:34:58.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:34:58.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:34:58.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:34:58.342 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:34:58.342 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:34:58.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:34:58.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:34:58.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:34:58.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:34:58.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:34:58.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:34:58.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:34:58.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:34:58.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:34:58.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:34:58.983 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:34:58.983 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr:7' 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:34:58.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:34:59.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-29T05:34:59.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-29T05:34:59.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-29T05:34:59.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:34:59.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:34:59.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-29T05:34:59.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-29T05:34:59.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977' 2024-03-29T05:34:59.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:34:59.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:35:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-29T05:35:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-29T05:35:00.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509455' 2024-03-29T05:35:00.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:35:00.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T05:35:00.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-03-29T05:35:00.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-03-29T05:35:00.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509455 3-85899345932' 2024-03-29T05:35:00.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:35:00.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T05:35:00.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182410 2024-03-29T05:35:00.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182410 2024-03-29T05:35:00.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509455 3-85899345932 4-107374182410' 2024-03-29T05:35:00.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:35:00.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T05:35:00.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-03-29T05:35:00.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-03-29T05:35:00.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509455 3-85899345932 4-107374182410 5-128849018887' 2024-03-29T05:35:00.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:35:00.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T05:35:00.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855366 2024-03-29T05:35:00.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855366 2024-03-29T05:35:00.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509455 3-85899345932 4-107374182410 5-128849018887 6-150323855366' 2024-03-29T05:35:00.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:35:00.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T05:35:00.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691843 2024-03-29T05:35:00.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691843 2024-03-29T05:35:00.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509455 3-85899345932 4-107374182410 5-128849018887 6-150323855366 7-171798691843' 2024-03-29T05:35:00.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:00.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-29T05:35:00.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:00.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:35:00.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-29T05:35:00.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:00.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-29T05:35:00.780 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836501 2024-03-29T05:35:00.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-29T05:35:00.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:35:01.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-29T05:35:01.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:35:02.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:35:02.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:35:02.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-29T05:35:02.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:02.397 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-29T05:35:02.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:02.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:35:02.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-29T05:35:02.400 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:02.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-29T05:35:02.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-29T05:35:02.402 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672977 2024-03-29T05:35:02.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:35:02.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-03-29T05:35:02.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:02.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-29T05:35:02.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:02.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:35:02.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-29T05:35:02.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:02.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-29T05:35:02.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-29T05:35:02.728 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509455 2024-03-29T05:35:02.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:35:03.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-29T05:35:03.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:03.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:03.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-03-29T05:35:03.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T05:35:03.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-03-29T05:35:03.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:03.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-03-29T05:35:03.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-03-29T05:35:03.055 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345932 2024-03-29T05:35:03.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T05:35:03.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345933 -lt 85899345932 2024-03-29T05:35:03.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:03.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182410 2024-03-29T05:35:03.397 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:03.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T05:35:03.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182410 2024-03-29T05:35:03.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:03.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182410 2024-03-29T05:35:03.400 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.4 seq 107374182410 2024-03-29T05:35:03.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-03-29T05:35:03.401 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T05:35:03.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182410 -lt 107374182410 2024-03-29T05:35:03.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:03.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-03-29T05:35:03.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:03.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T05:35:03.710 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-03-29T05:35:03.710 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:03.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-03-29T05:35:03.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-03-29T05:35:03.711 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.5 seq 128849018887 2024-03-29T05:35:03.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T05:35:04.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018887 2024-03-29T05:35:04.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:04.032 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855366 2024-03-29T05:35:04.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:04.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T05:35:04.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855366 2024-03-29T05:35:04.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:04.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855366 2024-03-29T05:35:04.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855366' 2024-03-29T05:35:04.037 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.6 seq 150323855366 2024-03-29T05:35:04.037 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T05:35:04.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855366 -lt 150323855366 2024-03-29T05:35:04.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:35:04.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:35:04.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691843 2024-03-29T05:35:04.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T05:35:04.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691843 2024-03-29T05:35:04.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:35:04.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691843 2024-03-29T05:35:04.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691843' 2024-03-29T05:35:04.383 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.7 seq 171798691843 2024-03-29T05:35:04.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T05:35:04.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691843 2024-03-29T05:35:04.710 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:35:04.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:35:04.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:35:05.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-03-29T05:35:05.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:35:05.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:35:05.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:35:05.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:35:05.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:35:05.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:35:05.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:35:05.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-03-29T05:35:05.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:35:05.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:35:05.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:35:05.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-03-29T05:35:05.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:35:05.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:35:05.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-03-29T05:35:05.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-03-29T05:35:05.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-03-29T05:35:06.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-03-29T05:35:06.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.36690 bs=1M count=50 2024-03-29T05:35:06.324 INFO:tasks.workunit.client.0.smithi123.stderr:50+0 records in 2024-03-29T05:35:06.324 INFO:tasks.workunit.client.0.smithi123.stderr:50+0 records out 2024-03-29T05:35:06.324 INFO:tasks.workunit.client.0.smithi123.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.16794 s, 312 MB/s 2024-03-29T05:35:06.324 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-03-29T05:35:06.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:06.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.36690 2024-03-29T05:35:08.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:08.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.36690 2024-03-29T05:35:09.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:09.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.36690 2024-03-29T05:35:11.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:11.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.36690 2024-03-29T05:35:13.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:13.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.36690 2024-03-29T05:35:15.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:15.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.36690 2024-03-29T05:35:17.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:17.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.36690 2024-03-29T05:35:18.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:18.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.36690 2024-03-29T05:35:20.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:20.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.36690 2024-03-29T05:35:22.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:22.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.36690 2024-03-29T05:35:24.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:24.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.36690 2024-03-29T05:35:26.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:26.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.36690 2024-03-29T05:35:28.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:28.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.36690 2024-03-29T05:35:29.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:29.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.36690 2024-03-29T05:35:31.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:31.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.36690 2024-03-29T05:35:33.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:33.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.36690 2024-03-29T05:35:35.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:35.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.36690 2024-03-29T05:35:37.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:37.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.36690 2024-03-29T05:35:39.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:39.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.36690 2024-03-29T05:35:40.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:40.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.36690 2024-03-29T05:35:43.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:43.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.36690 2024-03-29T05:35:44.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:44.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.36690 2024-03-29T05:35:46.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:46.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.36690 2024-03-29T05:35:48.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:48.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.36690 2024-03-29T05:35:50.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:50.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.36690 2024-03-29T05:35:52.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:52.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.36690 2024-03-29T05:35:53.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:53.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.36690 2024-03-29T05:35:55.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:55.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.36690 2024-03-29T05:35:57.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:57.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.36690 2024-03-29T05:35:59.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:35:59.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.36690 2024-03-29T05:36:01.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:01.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.36690 2024-03-29T05:36:03.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:03.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.36690 2024-03-29T05:36:05.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:05.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.36690 2024-03-29T05:36:06.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:06.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.36690 2024-03-29T05:36:08.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:08.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.36690 2024-03-29T05:36:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.36690 2024-03-29T05:36:12.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:12.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.36690 2024-03-29T05:36:14.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:14.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.36690 2024-03-29T05:36:16.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:16.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.36690 2024-03-29T05:36:18.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-29T05:36:18.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.36690 2024-03-29T05:36:19.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.36690 2024-03-29T05:36:19.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-03-29T05:36:20.303 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 3 2024-03-29T05:36:20.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-29T05:36:20.627 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:36:20.627 INFO:tasks.workunit.client.0.smithi123.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-29T05:34:59.327630+0000 46'26 46:56 [5] 5 [5] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.627 INFO:tasks.workunit.client.0.smithi123.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:59.327351+0000 46'13 46:43 [3] 3 [3] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.627 INFO:tasks.workunit.client.0.smithi123.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.285291+0000 46'13 46:31 [1] 1 [1] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.627 INFO:tasks.workunit.client.0.smithi123.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.142401+0000 46'13 46:34 [1] 1 [1] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.284099+0000 46'13 46:34 [1] 1 [1] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:34:58.283930+0000 0'0 46:21 [1] 1 [1] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-29T05:34:59.327298+0000 46'26 46:46 [5] 5 [5] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:59.217928+0000 46'13 46:34 [6] 6 [6] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.6 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:59.327014+0000 46'13 46:36 [6] 6 [6] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-29T05:34:58.799149+0000 46'39 46:61 [4] 4 [4] 4 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-29T05:34:59.326815+0000 46'52 46:74 [3] 3 [3] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-29T05:34:58.670008+0000 46'52 46:74 [4] 4 [4] 4 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.670584+0000 46'13 46:37 [0] 0 [0] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:34:59.327397+0000 0'0 46:19 [3] 3 [3] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:59.184201+0000 46'13 46:43 [5] 5 [5] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.845459+0000 46'13 46:31 [0] 0 [0] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.628 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.670552+0000 46'13 46:42 [7] 7 [7] 7 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.670147+0000 46'13 46:31 [7] 7 [7] 7 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.285049+0000 46'13 46:32 [1] 1 [1] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:59.327445+0000 46'13 46:32 [6] 6 [6] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:59.327008+0000 46'13 46:31 [5] 5 [5] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.671109+0000 46'13 46:32 [0] 0 [0] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-29T05:34:58.845138+0000 46'26 46:45 [0] 0 [0] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.799393+0000 46'13 46:30 [4] 4 [4] 4 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:34:58.670033+0000 0'0 46:18 [2] 2 [2] 2 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-29T05:34:58.284622+0000 46'39 46:58 [1] 1 [1] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:34:59.218209+0000 0'0 46:18 [6] 6 [6] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.845301+0000 46'13 46:31 [0] 0 [0] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:34:59.326984+0000 0'0 46:17 [5] 5 [5] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:59.184163+0000 46'13 46:30 [3] 3 [3] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.629 INFO:tasks.workunit.client.0.smithi123.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:34:58.285088+0000 0'0 46:18 [1] 1 [1] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.630 INFO:tasks.workunit.client.0.smithi123.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-29T05:34:58.845019+0000 46'13 46:31 [0] 0 [0] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:20.630 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:36:20.630 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:36:20.630 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:36:20.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-03-29T05:36:20.790 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:36:20.791 INFO:tasks.workunit.client.0.smithi123.stdout: "osd_recovery_max_active": "0" 2024-03-29T05:36:20.791 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:36:20.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-03-29T05:36:21.124 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:36:21.124 INFO:tasks.workunit.client.0.smithi123.stdout: "osd_recovery_max_active_hdd": "3" 2024-03-29T05:36:21.124 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:36:21.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-03-29T05:36:21.417 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:36:21.417 INFO:tasks.workunit.client.0.smithi123.stdout: "osd_recovery_max_active_ssd": "10" 2024-03-29T05:36:21.417 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:36:21.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-03-29T05:36:21.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-29T05:36:21.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-29T05:36:21.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-29T05:36:21.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-29T05:36:21.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-29T05:36:21.842 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:36:21.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-29T05:36:21.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-29T05:36:23.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-03-29T05:36:23.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-03-29T05:36:23.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-03-29T05:36:23.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-29T05:36:23.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-29T05:36:23.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-29T05:36:23.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-29T05:36:23.866 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-29T05:36:24.200 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:36:24.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-29T05:36:24.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-29T05:36:26.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-03-29T05:36:26.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-03-29T05:36:26.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-03-29T05:36:26.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-29T05:36:26.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-29T05:36:26.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-29T05:36:26.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-29T05:36:26.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-29T05:36:26.531 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:36:26.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-29T05:36:26.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-29T05:36:28.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-03-29T05:36:28.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-03-29T05:36:28.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-03-29T05:36:28.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-29T05:36:28.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-29T05:36:28.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-29T05:36:28.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-29T05:36:28.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-29T05:36:28.857 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:36:28.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-03-29T05:36:28.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-03-29T05:36:28.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-29T05:36:29.168 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-29T05:36:22.129381+0000 46'26 51:72 [5,0,2] 5 [5,0,2] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:24.276066+0000 46'13 51:59 [3,5,7] 3 [3,5,7] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-29T05:36:22.607529+0000 46'26 51:56 [1,3,0] 1 [1,3,0] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.550223+0000 46'13 51:46 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.759560+0000 46'13 51:46 [1,6,2] 1 [1,6,2] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:36:22.328330+0000 0'0 51:35 [1,4,5] 1 [1,4,5] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-29T05:36:22.761061+0000 46'26 51:61 [5,4,3] 5 [5,4,3] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.331086+0000 46'13 51:46 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.397325+0000 46'13 51:48 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-29T05:36:22.550183+0000 46'39 50:71 [4,2,6] 4 [4,2,6] 4 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.169 INFO:tasks.workunit.client.0.smithi123.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-29T05:36:22.760389+0000 46'52 51:88 [3,0,6] 3 [3,0,6] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-29T05:36:22.500340+0000 46'52 50:84 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.950449+0000 46'13 50:48 [0,7,1] 0 [0,7,1] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:36:22.332711+0000 0'0 51:35 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.129915+0000 46'13 51:59 [5,0,4] 5 [5,0,4] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.948144+0000 46'13 50:42 [0,3,4] 0 [0,3,4] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.761177+0000 46'13 51:57 [7,0,2] 7 [7,0,2] 7 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.759597+0000 46'13 51:44 [7,4,2] 7 [7,4,2] 7 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-29T05:36:22.334882+0000 46'13 51:46 [1,4,2] 1 [1,4,2] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.760031+0000 46'13 51:44 [6,0,3] 6 [6,0,3] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-29T05:36:22.083836+0000 46'13 51:47 [5,7,6] 5 [5,7,6] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.949635+0000 46'13 50:44 [0,1,3] 0 [0,1,3] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-29T05:36:22.550046+0000 46'26 50:56 [0,1,6] 0 [0,1,6] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.170 INFO:tasks.workunit.client.0.smithi123.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.397366+0000 46'13 50:40 [4,6,3] 4 [4,6,3] 4 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:36:22.763700+0000 0'0 50:30 [2,0,4] 2 [2,0,4] 2 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-29T05:36:22.607819+0000 46'39 51:70 [1,5,3] 1 [1,5,3] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:36:22.329111+0000 0'0 51:32 [6,0,7] 6 [6,0,7] 6 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.947706+0000 46'13 50:42 [0,4,5] 0 [0,4,5] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:36:21.987014+0000 0'0 51:33 [5,2,3] 5 [5,2,3] 5 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:23.161486+0000 46'13 51:43 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-29T05:36:22.333291+0000 0'0 51:32 [1,7,4] 1 [1,7,4] 1 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 periodic scrub scheduled @ 2024-03-30T05:34:57.262538+0000 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-29T05:36:22.950239+0000 46'13 50:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-29T05:34:57.262538+0000 0'0 2024-03-29T05:34:57.262538+0000 0 0 no scrub is scheduled 0 0 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:36:29.171 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:36:29.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-03-29T05:36:29.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-03-29T05:36:29.186 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-03-29T05:36:29.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-03-29T05:36:29.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.189 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-03-29T05:36:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-03-29T05:36:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58058"' 2024-03-29T05:36:29.190 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58058' 2024-03-29T05:36:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-03-29T05:36:29.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58061: /' 2024-03-29T05:36:29.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-03-29T05:36:29.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-03-29T05:36:29.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58065"' 2024-03-29T05:36:29.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58065' 2024-03-29T05:36:29.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-03-29T05:36:29.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58067: /' 2024-03-29T05:36:29.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-03-29T05:36:29.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-03-29T05:36:29.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58072"' 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58072' 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58075: /' 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58077"' 2024-03-29T05:36:29.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58077' 2024-03-29T05:36:29.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58081: /' 2024-03-29T05:36:29.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-03-29T05:36:29.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-03-29T05:36:29.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58083"' 2024-03-29T05:36:29.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58083' 2024-03-29T05:36:29.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-03-29T05:36:29.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-03-29T05:36:29.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-03-29T05:36:29.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58087"' 2024-03-29T05:36:29.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58087' 2024-03-29T05:36:29.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-03-29T05:36:29.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-03-29T05:36:29.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-03-29T05:36:29.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58095"' 2024-03-29T05:36:29.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58095' 2024-03-29T05:36:29.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58088: /' 2024-03-29T05:36:29.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-03-29T05:36:29.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58094: /' 2024-03-29T05:36:29.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-03-29T05:36:29.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58101: /' 2024-03-29T05:36:29.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-03-29T05:36:29.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58102"' 2024-03-29T05:36:29.229 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58102' 2024-03-29T05:36:29.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-03-29T05:36:29.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58107: /' 2024-03-29T05:36:29.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58109"' 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58109' 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-03-29T05:36:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-03-29T05:36:29.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-03-29T05:36:29.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-03-29T05:36:29.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58117"' 2024-03-29T05:36:29.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58117' 2024-03-29T05:36:29.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-03-29T05:36:29.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-03-29T05:36:29.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-03-29T05:36:29.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58124"' 2024-03-29T05:36:29.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58124' 2024-03-29T05:36:29.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58112: /' 2024-03-29T05:36:29.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58122: /' 2024-03-29T05:36:29.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58131: /' 2024-03-29T05:36:29.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-03-29T05:36:29.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-03-29T05:36:29.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-03-29T05:36:29.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58133"' 2024-03-29T05:36:29.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58133' 2024-03-29T05:36:29.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.292 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-03-29T05:36:29.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-03-29T05:36:29.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58137"' 2024-03-29T05:36:29.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58137' 2024-03-29T05:36:29.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58140: /' 2024-03-29T05:36:29.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-03-29T05:36:29.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-03-29T05:36:29.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-03-29T05:36:29.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58142: /' 2024-03-29T05:36:29.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-03-29T05:36:29.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58145"' 2024-03-29T05:36:29.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58145' 2024-03-29T05:36:29.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-03-29T05:36:29.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-03-29T05:36:29.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58153"' 2024-03-29T05:36:29.338 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58153' 2024-03-29T05:36:29.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-03-29T05:36:29.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58156: /' 2024-03-29T05:36:29.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58148: /' 2024-03-29T05:36:29.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-03-29T05:36:29.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-03-29T05:36:29.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-03-29T05:36:29.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58159"' 2024-03-29T05:36:29.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58159' 2024-03-29T05:36:29.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-03-29T05:36:29.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-03-29T05:36:29.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58167"' 2024-03-29T05:36:29.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58167' 2024-03-29T05:36:29.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-03-29T05:36:29.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58166: /' 2024-03-29T05:36:29.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-03-29T05:36:29.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-03-29T05:36:29.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58170"' 2024-03-29T05:36:29.397 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58170' 2024-03-29T05:36:29.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-03-29T05:36:29.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-03-29T05:36:29.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58178"' 2024-03-29T05:36:29.413 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58178' 2024-03-29T05:36:29.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-03-29T05:36:29.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58173: /' 2024-03-29T05:36:29.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-03-29T05:36:29.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58177: /' 2024-03-29T05:36:29.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-03-29T05:36:29.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-03-29T05:36:29.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58185"' 2024-03-29T05:36:29.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58185' 2024-03-29T05:36:29.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-03-29T05:36:29.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-03-29T05:36:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-03-29T05:36:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58194"' 2024-03-29T05:36:29.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58194' 2024-03-29T05:36:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-03-29T05:36:29.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58188: /' 2024-03-29T05:36:29.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58190: /' 2024-03-29T05:36:29.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-03-29T05:36:29.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-03-29T05:36:29.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58200"' 2024-03-29T05:36:29.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58200' 2024-03-29T05:36:29.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-03-29T05:36:29.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-03-29T05:36:29.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58204"' 2024-03-29T05:36:29.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58204' 2024-03-29T05:36:29.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-03-29T05:36:29.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58203: /' 2024-03-29T05:36:29.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-03-29T05:36:29.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-03-29T05:36:29.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58213"' 2024-03-29T05:36:29.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58213' 2024-03-29T05:36:29.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-03-29T05:36:29.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58210: /' 2024-03-29T05:36:29.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-03-29T05:36:29.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-03-29T05:36:29.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-03-29T05:36:29.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58229"' 2024-03-29T05:36:29.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58229' 2024-03-29T05:36:29.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-03-29T05:36:29.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58212: /' 2024-03-29T05:36:29.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58228: /' 2024-03-29T05:36:29.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-03-29T05:36:29.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-03-29T05:36:29.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58243"' 2024-03-29T05:36:29.522 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58243' 2024-03-29T05:36:29.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-03-29T05:36:29.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58242: /' 2024-03-29T05:36:29.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-03-29T05:36:29.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-03-29T05:36:29.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58247"' 2024-03-29T05:36:29.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58247' 2024-03-29T05:36:29.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-03-29T05:36:29.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-03-29T05:36:29.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58253"' 2024-03-29T05:36:29.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58253' 2024-03-29T05:36:29.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58252: /' 2024-03-29T05:36:29.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58256: /' 2024-03-29T05:36:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-03-29T05:36:29.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-03-29T05:36:29.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-03-29T05:36:29.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58270"' 2024-03-29T05:36:29.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58270' 2024-03-29T05:36:29.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-03-29T05:36:29.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58266: /' 2024-03-29T05:36:29.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-03-29T05:36:29.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-03-29T05:36:29.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-03-29T05:36:29.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58288"' 2024-03-29T05:36:29.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58288' 2024-03-29T05:36:29.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58283: /' 2024-03-29T05:36:29.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-03-29T05:36:29.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-03-29T05:36:29.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-03-29T05:36:29.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58293"' 2024-03-29T05:36:29.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58293' 2024-03-29T05:36:29.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-29T05:36:29.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58292: /' 2024-03-29T05:36:29.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-03-29T05:36:29.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-03-29T05:36:29.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 58300"' 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 58300' 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 58058 58065 58072 58077 58083 58087 58095 58102 58109 58117 58124 58133 58137 58145 58153 58159 58167 58170 58178 58185 58194 58200 58204 58213 58229 58243 58247 58253 58270 58288 58293 58300' 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:29.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58058 2024-03-29T05:36:29.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-03-29T05:36:29.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58309: /' 2024-03-29T05:36:29.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/58315: /' 2024-03-29T05:36:29.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-03-29T05:36:36.733 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-03-29T05:36:36.733 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: instructing pg 1.2 on osd.0 to scrub 2024-03-29T05:36:36.734 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: dumped pgs 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:36.735 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:36.736 INFO:tasks.workunit.client.0.smithi123.stderr:58075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-29T05:36:36.736 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:36.736 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:36.736 INFO:tasks.workunit.client.0.smithi123.stderr:58075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:36.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: instructing pg 1.4 on osd.3 to scrub 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: dumped pgs 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:58088: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:36.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:37.169 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: instructing pg 1.b on osd.1 to scrub 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: dumped pgs 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:37.170 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:58140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:37.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:37.235 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: instructing pg 1.5 on osd.4 to scrub 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: dumped pgs 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:58094: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.011 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: instructing pg 1.7 on osd.6 to scrub 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: dumped pgs 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.012 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:58107: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.065 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: instructing pg 1.d on osd.7 to scrub 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: dumped pgs 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.066 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:58148: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-03-29T05:36:38.322 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: instructing pg 1.9 on osd.1 to scrub 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: dumped pgs 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.323 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:36:33.479055+0000 '>' '' 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:58122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.325 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-03-29T05:36:38.326 INFO:tasks.workunit.client.0.smithi123.stderr:58061: instructing pg 1.0 on osd.7 to scrub 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: dumped pgs 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:36:38.327 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:36:38.328 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.328 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.328 INFO:tasks.workunit.client.0.smithi123.stderr:58061: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:36:38.328 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.328 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.328 INFO:tasks.workunit.client.0.smithi123.stderr:58061: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58065 2024-03-29T05:36:38.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-03-29T05:36:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: instructing pg 1.1 on osd.5 to scrub 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: dumped pgs 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:58067: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58072 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58077 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: instructing pg 1.10 on osd.5 to scrub 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: dumped pgs 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.430 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.430 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.430 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-03-29T05:36:38.430 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-29T05:36:38.430 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.430 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58173: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stat58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.448 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: instructing pg 1.e on osd.1 to scrub 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: dumped pgs 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.449 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.450 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.450 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.450 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-03-29T05:36:38.450 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-29T05:36:38.450 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.450 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58156: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.465 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: instructing pg 1.6 on osd.6 to scrub 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.466 INFO:tasks.workunit.client.0.smithi123.stderr:58101: dumped pgs 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.467 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.468 INFO:tasks.workunit.client.0.smithi123.stderr:58101: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.477 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-03-29T05:36:38.477 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-03-29T05:36:38.477 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-29T05:36:38.477 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.477 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-29T05:36:38.477 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.477 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: instructing pg 1.a on osd.1 to scrub 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: dumped pgs 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.478 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-29T05:36:38.479 INFO:tasks.workunit.client.0.smithi123.stderr:58131: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.480 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.480 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.480 INFO:tasks.workunit.client.0.smithi123.stderr:58131: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.601 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-03-29T05:36:38.601 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-03-29T05:36:38.601 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-29T05:36:38.601 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.601 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-29T05:36:38.601 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.601 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: instructing pg 1.1d on osd.1 to scrub 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.602 INFO:tasks.workunit.client.0.smithi123.stderr:58292: dumped pgs 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-29T05:36:38.603 INFO:tasks.workunit.client.0.smithi123.stderr:58292: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.604 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.604 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.604 INFO:tasks.workunit.client.0.smithi123.stderr:58292: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.694 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: instructing pg 1.18 on osd.5 to scrub 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: dumped pgs 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.695 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:36:34.219537+0000 '>' '' 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:58242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.709 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-03-29T05:36:38.710 INFO:tasks.workunit.client.0.smithi123.stderr:58203: instructing pg 1.14 on osd.2 to scrub 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: dumped pgs 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-03-29T05:36:38.711 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-29T05:36:38.712 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.712 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.712 INFO:tasks.workunit.client.0.smithi123.stderr:58203: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-29T05:36:38.712 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.712 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.712 INFO:tasks.workunit.client.0.smithi123.stderr:58203: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.716 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-03-29T05:36:38.716 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-03-29T05:36:38.716 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-29T05:36:38.716 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.716 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-29T05:36:38.716 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.716 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: instructing pg 1.1b on osd.0 to scrub 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: dumped pgs 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.717 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.718 INFO:tasks.workunit.client.0.smithi123.stderr:58266: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.757 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-03-29T05:36:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:58166: instructing pg 1.f on osd.6 to scrub 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: dumped pgs 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.759 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-29T05:36:38.760 INFO:tasks.workunit.client.0.smithi123.stderr:58166: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.760 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.760 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.760 INFO:tasks.workunit.client.0.smithi123.stderr:58166: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: instructing pg 1.3 on osd.4 to scrub 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: dumped pgs 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.779 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.779 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.779 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.779 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.780 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T05:36:38.780 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T05:36:38.780 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.780 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.780 INFO:tasks.workunit.client.0.smithi123.stderr:58081: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:58081: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58083 2024-03-29T05:36:38.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58087 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58095 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58102 2024-03-29T05:36:38.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58109 2024-03-29T05:36:38.816 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-03-29T05:36:38.816 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: instructing pg 1.1c on osd.0 to scrub 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.817 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: dumped pgs 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-29T05:36:38.818 INFO:tasks.workunit.client.0.smithi123.stderr:58283: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.819 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.819 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.819 INFO:tasks.workunit.client.0.smithi123.stderr:58283: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.842 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-03-29T05:36:38.842 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: instructing pg 1.11 on osd.0 to scrub 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.843 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: dumped pgs 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-29T05:36:38.844 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.845 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.845 INFO:tasks.workunit.client.0.smithi123.stderr:58177: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: instructing pg 1.1e on osd.3 to scrub 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: dumped pgs 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:58309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:58309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.933 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: instructing pg 1.12 on osd.0 to scrub 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: dumped pgs 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.934 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.935 INFO:tasks.workunit.client.0.smithi123.stderr:58188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.963 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-03-29T05:36:38.963 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-03-29T05:36:38.963 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-29T05:36:38.963 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.963 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.963 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: instructing pg 1.8 on osd.5 to scrub 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: dumped pgs 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:58112: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58117 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58124 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58133 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:38.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58137 2024-03-29T05:36:39.011 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-03-29T05:36:39.011 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-03-29T05:36:39.011 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-29T05:36:39.011 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.011 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: instructing pg 1.1a on osd.1 to scrub 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.012 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: dumped pgs 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.013 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:58256: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:36:34.459792+0000 '>' '' 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:58256: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.134 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-03-29T05:36:39.134 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-03-29T05:36:39.134 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-29T05:36:39.134 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.134 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: instructing pg 1.1f on osd.5 to scrub 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.135 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: dumped pgs 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.136 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.137 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.137 INFO:tasks.workunit.client.0.smithi123.stderr:58315: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.159 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: instructing pg 1.13 on osd.4 to scrub 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: dumped pgs 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.160 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.161 INFO:tasks.workunit.client.0.smithi123.stderr:58190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.199 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: instructing pg 1.17 on osd.0 to scrub 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: dumped pgs 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-03-29T05:36:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-29T05:36:39.202 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.202 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.202 INFO:tasks.workunit.client.0.smithi123.stderr:58228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-29T05:36:39.202 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.202 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.202 INFO:tasks.workunit.client.0.smithi123.stderr:58228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-03-29T05:36:39.208 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: instructing pg 1.16 on osd.6 to scrub 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: dumped pgs 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.209 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:58212: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.261 INFO:tasks.workunit.client.0.smithi123.stderr:lect(.pgid=="1.e") | .last_scrub_stamp' 2024-03-29T05:36:39.262 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.262 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.262 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-29T05:36:39.262 INFO:tasks.workunit.client.0.smithi123.stderr:58156: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-29T05:36:39.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-29T05:36:39.314 INFO:tasks.workunit.client.0.smithi123.stderr:s | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-29T05:36:39.314 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.314 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.314 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-29T05:36:39.314 INFO:tasks.workunit.client.0.smithi123.stderr:58173: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-29T05:36:39.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-29T05:36:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-03-29T05:36:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:58142: instructing pg 1.c on osd.3 to scrub 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: dumped pgs 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:58142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:36:34.781522+0000 '>' '' 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:58142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58145 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58153 2024-03-29T05:36:39.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-29T05:36:39.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-29T05:36:39.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58159 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58167 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58170 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58178 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58185 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58194 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58200 2024-03-29T05:36:39.356 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: instructing pg 1.15 on osd.1 to scrub 2024-03-29T05:36:39.357 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: dumped pgs 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-29T05:36:39.358 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:58210: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:58210: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58204 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58213 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58229 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58243 2024-03-29T05:36:39.375 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-03-29T05:36:39.375 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-03-29T05:36:39.375 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: instructing pg 1.19 on osd.3 to scrub 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-29T05:36:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: dumped pgs 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-29T05:34:57.262538+0000 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-29T05:34:57.262538+0000 '>' '' 2024-03-29T05:36:39.377 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:58252: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58247 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58253 2024-03-29T05:36:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58270 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58288 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58293 2024-03-29T05:36:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 58300 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-03-29T05:36:39.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.5.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/osd.2.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/osd.3.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/osd.7.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/osd.4.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/mon.a.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/osd.1.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/osd.0.pid 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:td/osd-recovery-scrub/osd.6.pid' 2024-03-29T05:36:39.381 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.6.pid 2024-03-29T05:36:39.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53088 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:48958 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:50336 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:45263 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:55840 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:51712 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:45179 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:47582 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:46206 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:54464' 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53088 48958 50336 45263 55840 51712 45179 47582 46206 54464 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:36:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:36:44.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:36:44.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-03-29T05:36:44.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-03-29T05:36:44.694 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-03-29T05:36:44.696 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-03-29T05:36:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-03-29T05:36:44.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-03-29T05:36:44.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-03-29T05:36:44.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-03-29T05:36:44.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-03-29T05:36:44.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-03-29T05:36:44.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-03-29T05:36:44.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-03-29T05:36:44.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-03-29T05:36:44.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-03-29T05:36:44.738 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-03-29T05:36:44.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-03-29T05:36:44.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-03-29T05:36:44.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-03-29T05:36:44.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-03-29T05:36:44.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-03-29T05:36:44.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-03-29T05:36:44.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-03-29T05:36:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-03-29T05:36:44.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T05:36:44.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-03-29T05:36:44.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-03-29T05:36:44.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-03-29T05:36:44.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-29T05:36:44.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:36:44.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-29T05:36:44.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:36:44.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:36:44.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:36:44.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:36:44.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:36:44.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:36:44.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:36:44.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:36:44.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:36:44.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:36:44.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:36:44.789 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:36:44.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:44.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:36:44.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:36:44.790 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:44.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:36:44.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:36:44.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-29T05:36:44.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:36:44.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:44.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:36:44.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36690 2024-03-29T05:36:44.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:36:44.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:36:44.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-03-29T05:36:44.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-03-29T05:36:44.821 INFO:tasks.workunit.client.0.smithi123.stdout:TEST PASSED 2024-03-29T05:36:44.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-03-29T05:36:44.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-29T05:36:44.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-03-29T05:36:44.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:36:44.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:36:44.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:36:44.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:36:44.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:36:44.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:36:44.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:36:44.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:36:44.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:44.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:36:44.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:36:44.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:44.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:36:44.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-29T05:36:44.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-29T05:36:44.830 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:36:44.830 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:44.830 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.36690 2024-03-29T05:36:44.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.36690 2024-03-29T05:36:44.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:36:44.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:36:44.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-29T05:36:44.832 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-29T05:36:44.832 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-29T05:36:44.885 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-03-29T05:36:44.886 DEBUG:teuthology.orchestra.run.smithi123:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-03-29T05:36:44.938 INFO:tasks.workunit.client.0.smithi123.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-03-29T05:36:44.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-03-29T05:36:44.945 INFO:tasks.workunit.client.0.smithi123.stdout:This test is disabled 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:36:44.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:36:44.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:36:44.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:36:44.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:36:44.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:36:44.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:36:44.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:36:44.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:36:44.949 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:36:44.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:44.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:36:44.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:36:44.951 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:44.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:36:44.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-29T05:36:44.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-03-29T05:36:44.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:36:44.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:44.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61520 2024-03-29T05:36:44.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61520 2024-03-29T05:36:44.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:36:44.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:36:44.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-29T05:36:44.956 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-29T05:36:44.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-29T05:36:45.010 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-03-29T05:36:45.011 DEBUG:teuthology.orchestra.run.smithi123:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-03-29T05:36:45.061 INFO:tasks.workunit.client.0.smithi123.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ TIMEOUT=300 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ MAX_TIMEOUT=15 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ PG_NUM=4 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ TMPDIR=/tmp 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ type xmlstarlet 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:++ XMLSTARLET=xmlstarlet 2024-03-29T05:36:45.062 INFO:tasks.workunit.client.0.smithi123.stderr:+++ uname 2024-03-29T05:36:45.063 INFO:tasks.workunit.client.0.smithi123.stderr:++ '[' Linux = FreeBSD ']' 2024-03-29T05:36:45.063 INFO:tasks.workunit.client.0.smithi123.stderr:++ SED=sed 2024-03-29T05:36:45.063 INFO:tasks.workunit.client.0.smithi123.stderr:++ AWK=awk 2024-03-29T05:36:45.063 INFO:tasks.workunit.client.0.smithi123.stderr:+++ stty -a 2024-03-29T05:36:45.064 INFO:tasks.workunit.client.0.smithi123.stderr:+++ head -1 2024-03-29T05:36:45.064 INFO:tasks.workunit.client.0.smithi123.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-03-29T05:36:45.064 INFO:tasks.workunit.client.0.smithi123.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-29T05:36:45.064 INFO:tasks.workunit.client.0.smithi123.stderr:++ termwidth= 2024-03-29T05:36:45.064 INFO:tasks.workunit.client.0.smithi123.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-03-29T05:36:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:++ DIFFCOLOPTS='-y ' 2024-03-29T05:36:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:++ KERNCORE=kernel.core_pattern 2024-03-29T05:36:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:++ EXTRA_OPTS= 2024-03-29T05:36:45.069 INFO:tasks.workunit.client.0.smithi123.stderr:++ test '' = TESTS 2024-03-29T05:36:45.069 INFO:tasks.workunit.client.0.smithi123.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-03-29T05:36:45.070 INFO:tasks.workunit.client.0.smithi123.stderr:++ uname 2024-03-29T05:36:45.071 INFO:tasks.workunit.client.0.smithi123.stderr:+ '[' Linux = FreeBSD ']' 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr:+ use_ec_overwrite=true 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr:+ getjson=no 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr:+ jqfilter='def walk(f): 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T05:36:45.072 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T05:36:45.073 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T05:36:45.073 INFO:tasks.workunit.client.0.smithi123.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:+ main osd-scrub-repair 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:+ local dir=td/osd-scrub-repair 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:+ shift 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:+ shopt -s -o xtrace 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-29T05:36:45.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-03-29T05:36:45.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-03-29T05:36:45.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none ' 2024-03-29T05:36:45.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-03-29T05:36:45.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-03-29T05:36:45.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-29T05:36:45.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-03-29T05:36:45.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-29T05:36:45.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-03-29T05:36:45.085 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_auto_repair_bluestore_basic 2024-03-29T05:36:45.085 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_auto_repair_bluestore_failed 2024-03-29T05:36:45.085 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-03-29T05:36:45.085 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_auto_repair_bluestore_scrub 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_auto_repair_bluestore_tag 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_auto_repair_erasure_coded_appends 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_and_repair_lrc_appends 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_and_repair_replicated 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_scrub_erasure_appends 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_scrub_replicated 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_corrupt_snapset_scrub_rep 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_list_missing_erasure_coded_appends 2024-03-29T05:36:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_list_missing_erasure_coded_overwrites 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_periodic_scrub_replicated 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_repair_stats 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_repair_stats_ec 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_request_scrub_priority 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_scrub_warning 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_skip_non_repair_during_recovery 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_unfound_erasure_coded_appends 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:TEST_unfound_erasure_coded_overwrites' 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:36:45.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:36:45.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:36:45.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:36:45.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:36:45.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:36:45.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:36:45.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:36:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:36:45.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:36:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:36:45.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:36:45.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:36:45.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:36:45.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:36:45.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:45.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:36:45.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:36:45.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:36:45.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:36:45.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:36:45.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:36:45.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:36:45.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:45.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:36:45.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:36:45.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:36:45.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:36:45.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:36:45.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:36:45.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:45.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:36:45.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:36:45.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:36:45.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:36:45.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-29T05:36:45.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:36:45.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:36:45.518 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:36:45.518 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:36:45.518 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:45.519 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:36:45.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:36:45.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-29T05:36:45.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:36:45.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:36:45.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:36:45.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:36:45.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:36:45.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:36:45.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:36:45.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:36:45.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:36:45.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:36:45.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:36:45.863 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:36:45.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:36:45.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:36:45.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-03-29T05:36:45.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:36:45.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:36:45.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:36:45.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:36:45.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:36:45.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:36:46.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:36:46.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:36:46.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:36:46.837 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:36:46.837 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:46.837 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:36:46.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:36:46.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:36:46.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:36:46.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-29T05:36:46.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:36:46.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:36:46.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:36:46.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:36:46.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:36:46.878 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:36:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-29T05:36:46.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:36:46.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:36:46.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7728b29e-c567-4913-8ca1-a92e31f63586 2024-03-29T05:36:46.880 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 7728b29e-c567-4913-8ca1-a92e31f63586 2024-03-29T05:36:46.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7728b29e-c567-4913-8ca1-a92e31f63586' 2024-03-29T05:36:46.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:36:46.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuUwZmSSKcNRAALqteV8jm2sz+moecyJlsZA== 2024-03-29T05:36:46.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuUwZmSSKcNRAALqteV8jm2sz+moecyJlsZA=="}' 2024-03-29T05:36:46.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7728b29e-c567-4913-8ca1-a92e31f63586 -i td/osd-scrub-repair/0/new.json 2024-03-29T05:36:47.148 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:36:47.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:36:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBuUwZmSSKcNRAALqteV8jm2sz+moecyJlsZA== --osd-uuid 7728b29e-c567-4913-8ca1-a92e31f63586 2024-03-29T05:36:47.199 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:47.196+0000 7f0e048f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:36:47.201 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:47.200+0000 7f0e048f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:36:47.202 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:47.200+0000 7f0e048f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:36:47.203 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:47.200+0000 7f0e048f07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:36:47.203 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:47.200+0000 7f0e048f07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:36:47.203 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:47.200+0000 7f0e048f07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:36:47.282 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:47.280+0000 7f0e048f07c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:36:53.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:36:53.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:36:53.778 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:36:53.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:36:53.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:36:54.155 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:36:54.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:36:54.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-29T05:36:54.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:36:54.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:36:54.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:36:54.188 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:54.188+0000 7f79d89fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:36:54.220 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:54.220+0000 7f79d89fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:36:54.222 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:54.220+0000 7f79d89fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:36:54.487 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:36:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:36:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:36:54.777 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:54.776+0000 7f79d89fe7c0 -1 Falling back to public interface 2024-03-29T05:36:54.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:36:55.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:36:55.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:36:55.820 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:36:55.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:36:55.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:36:55.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:36:56.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:36:57.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:36:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:36:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:36:57.122 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:36:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:36:57.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:36:57.346 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:36:57.344+0000 7f79d89fe7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:36:57.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:36:58.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:36:58.459 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:36:58.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:36:58.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:36:58.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:36:58.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:36:58.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:36:59.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:36:59.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:36:59.779 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:36:59.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:36:59.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:36:59.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:00.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:01.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:01.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:01.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:37:01.201 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:37:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:01.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:02.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:02.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:02.524 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:37:02.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:37:02.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:02.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:02.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:03.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:03.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:03.845 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:37:03.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:37:03.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:03.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:05.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:05.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:05.350 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:37:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:37:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:05.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:06.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:06.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:06.665 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:37:06.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:37:06.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:06.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:07.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:08.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:08.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:08.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:37:08.006 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:37:08.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:08.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:08.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:09.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:09.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:09.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:37:09.339 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:37:09.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:09.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:09.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:10.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:10.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:10.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:37:10.648 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:37:10.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:10.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:11.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:12.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:12.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:12.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:37:12.235 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:37:12.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:12.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:12.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:13.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:13.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:13.862 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:37:13.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:37:13.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:13.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:14.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:15.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:15.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:15.692 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:37:15.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:37:15.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:15.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:16.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:17.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:17.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:17.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:37:17.668 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:37:17.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:17.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:18.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:19.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:19.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:19.463 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:37:19.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:37:19.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:19.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:19.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:20.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:20.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:20.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:37:20.939 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:37:20.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:20.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:21.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:22.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:22.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:22.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:37:22.661 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:37:22.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:22.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:37:22.980 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3845352929,v1:127.0.0.1:6803/3845352929] [v2:127.0.0.1:6804/3845352929,v1:127.0.0.1:6805/3845352929] exists,up 7728b29e-c567-4913-8ca1-a92e31f63586 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:37:22.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:37:22.985 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-29T05:37:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:37:22.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:37:22.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c6b88e6-f22e-4e90-9307-358c1eee5787 2024-03-29T05:37:22.987 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 9c6b88e6-f22e-4e90-9307-358c1eee5787 2024-03-29T05:37:22.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9c6b88e6-f22e-4e90-9307-358c1eee5787' 2024-03-29T05:37:22.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:37:23.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTUwZmIylUABAAo/iNonTwJhQL0pMcr8cBag== 2024-03-29T05:37:23.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTUwZmIylUABAAo/iNonTwJhQL0pMcr8cBag=="}' 2024-03-29T05:37:23.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c6b88e6-f22e-4e90-9307-358c1eee5787 -i td/osd-scrub-repair/1/new.json 2024-03-29T05:37:23.374 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:37:23.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:37:23.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQCTUwZmIylUABAAo/iNonTwJhQL0pMcr8cBag== --osd-uuid 9c6b88e6-f22e-4e90-9307-358c1eee5787 2024-03-29T05:37:23.431 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:23.432+0000 7fb29a2177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:37:23.484 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:23.484+0000 7fb29a2177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:37:23.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:23.484+0000 7fb29a2177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:37:23.487 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:23.488+0000 7fb29a2177c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:37:23.487 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:23.488+0000 7fb29a2177c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:37:23.487 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:23.488+0000 7fb29a2177c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:37:23.557 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:23.560+0000 7fb29a2177c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:37:30.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:37:30.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:37:30.070 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:37:30.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:37:30.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:37:30.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:37:30.453 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:37:30.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-29T05:37:30.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:37:30.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:37:30.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:37:30.480 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:30.479+0000 7f60bb3897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:37:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:30.479+0000 7f60bb3897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:37:30.484 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:30.483+0000 7f60bb3897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:37:30.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:37:30.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:37:30.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:37:30.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:37:30.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:37:30.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:30.808 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:37:30.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:37:30.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:30.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:31.025 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:31.023+0000 7f60bb3897c0 -1 Falling back to public interface 2024-03-29T05:37:31.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:32.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:32.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:32.148 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:37:32.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:37:32.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:32.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:32.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:33.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:33.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:33.449 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:37:33.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:37:33.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:33.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:33.683 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:37:33.683+0000 7f60bb3897c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:37:33.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:37:34.797 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:37:34.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:34.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:35.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:36.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:36.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:36.103 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:37:36.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:37:36.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:36.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:36.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:37.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:37.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:37.430 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:37:37.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:37:37.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:37.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:37.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:38.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:38.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:38.883 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:37:38.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:37:38.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:38.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:40.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:40.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:40.203 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:37:40.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:37:40.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:40.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:40.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:41.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:41.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:41.577 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:37:41.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:37:41.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:41.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:41.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:42.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:42.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:42.891 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:37:42.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:37:42.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:42.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:43.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:44.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:44.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:44.208 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:37:44.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:37:44.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:44.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:44.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:45.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:45.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:45.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:37:45.575 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:37:45.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:45.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:45.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:46.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:46.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:46.890 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:37:46.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:37:46.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:46.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:47.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:48.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:48.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:48.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:37:48.207 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:37:48.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:48.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:48.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:49.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:49.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:49.893 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:37:49.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:37:49.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:49.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:50.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:51.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:51.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:51.769 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:37:51.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:37:51.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:51.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:52.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:53.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:53.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:53.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:37:53.629 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:37:53.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:53.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:54.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:55.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:55.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:55.365 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:37:55.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:37:55.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:55.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:55.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:56.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:56.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:56.847 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:37:56.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:37:56.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:56.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:57.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:58.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:58.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:58.342 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:37:58.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:37:58.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:58.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:37:58.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:37:59.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:37:59.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:37:59.669 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:37:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:37:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:37:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:38:00.019 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2121001166,v1:127.0.0.1:6811/2121001166] [v2:127.0.0.1:6812/2121001166,v1:127.0.0.1:6813/2121001166] exists,up 9c6b88e6-f22e-4e90-9307-358c1eee5787 2024-03-29T05:38:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:38:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:38:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:38:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-03-29T05:38:00.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T05:38:00.313 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T05:38:00.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T05:38:00.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T05:38:00.712 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T05:38:00.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:38:01.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T05:38:04.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-03-29T05:38:04.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:38:04.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:38:04.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:38:04.223 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:38:04.223 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:38:04.223 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:38:04.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:38:04.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:38:04.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:38:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:38:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:38:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:38:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:38:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:38:04.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:38:04.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:38:04.661 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T05:38:04.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:38:04.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:38:04.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:38:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-03-29T05:38:04.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-03-29T05:38:04.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-03-29T05:38:04.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:38:04.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:38:05.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-29T05:38:05.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-29T05:38:05.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672967' 2024-03-29T05:38:05.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:38:05.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-03-29T05:38:05.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:38:05.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:38:05.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-03-29T05:38:05.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:38:05.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-03-29T05:38:05.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-03-29T05:38:05.004 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836491 2024-03-29T05:38:05.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:38:05.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-03-29T05:38:05.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:38:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:38:06.303 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:38:06.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-03-29T05:38:06.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:38:06.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-29T05:38:06.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:38:06.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:38:06.612 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-29T05:38:06.612 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:38:06.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-29T05:38:06.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-29T05:38:06.614 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672967 2024-03-29T05:38:06.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:38:06.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-03-29T05:38:06.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:38:06.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:38:06.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:38:07.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T05:38:07.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:38:07.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:38:07.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:38:07.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:38:07.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:38:07.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:38:07.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:38:07.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:38:07.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:38:07.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:38:07.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:38:08.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T05:38:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T05:38:08.541 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T05:38:08.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T05:38:08.985 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T05:38:09.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T05:38:09.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T05:38:09.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T05:38:09.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-03-29T05:38:09.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-29T05:38:09.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:38:09.301 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-29T05:38:09.301 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-29T05:38:09.301 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:38:09.301 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-29T05:38:09.302 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:38:09.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:38:09.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-29T05:38:09.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:38:10.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-29T05:38:10.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:38:10.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:38:10.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:38:10.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:38:10.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:38:10.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:38:10.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:38:11.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-29T05:38:12.747 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-29T05:38:13.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:38:13.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:38:13.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:38:13.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:38:13.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:38:13.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:38:13.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:38:13.287 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:38:13.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:38:13.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:38:13.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:38:13.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:38:13.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:38:13.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:38:13.320 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:13.319+0000 7fa594bcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:13.323 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:13.323+0000 7fa594bcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:13.323+0000 7fa594bcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:13.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:38:13.857 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:13.855+0000 7fa594bcd7c0 -1 Falling back to public interface 2024-03-29T05:38:13.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:38:14.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:38:14.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:14.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:38:14.952 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:38:14.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:14.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:38:15.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:38:16.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:38:16.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:16.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:38:16.269 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:38:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:16.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:38:16.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:38:16.652 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:16.651+0000 7fa594bcd7c0 -1 osd.0 21 log_to_monitors true 2024-03-29T05:38:17.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:38:17.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:17.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:38:17.582 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:38:17.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:17.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:38:17.953 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4704904,v1:127.0.0.1:6803/4704904] [v2:127.0.0.1:6804/4704904,v1:127.0.0.1:6805/4704904] exists,up 7728b29e-c567-4913-8ca1-a92e31f63586 2024-03-29T05:38:17.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:38:17.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:38:17.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:38:17.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:38:17.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:38:17.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:38:17.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:38:17.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:38:17.954 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:38:17.954 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:38:17.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:38:17.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:38:17.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:38:18.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:38:18.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:38:18.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:38:18.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:38:18.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:38:18.062 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:38:18.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:38:18.382 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T05:38:18.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:38:18.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:38:18.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:38:18.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-29T05:38:18.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-29T05:38:18.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-29T05:38:18.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:38:18.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:38:18.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-29T05:38:18.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-29T05:38:18.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672971' 2024-03-29T05:38:18.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:38:18.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:38:18.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-29T05:38:18.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:38:18.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-29T05:38:18.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:38:18.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-29T05:38:18.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-29T05:38:18.688 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 107374182402 2024-03-29T05:38:18.688 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:38:18.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-29T05:38:18.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:38:19.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:38:19.986 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:38:20.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-29T05:38:20.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:38:20.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:38:20.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-29T05:38:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:38:20.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-29T05:38:20.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:38:20.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-29T05:38:20.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-29T05:38:20.294 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672971 2024-03-29T05:38:20.294 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:38:20.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-29T05:38:20.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:38:20.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:38:20.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:38:21.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T05:38:21.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:38:21.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:38:21.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:38:21.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:38:21.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:38:21.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:38:21.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:38:21.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:38:21.345 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:38:21.345 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:38:21.345 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:38:21.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T05:38:21.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:38:21.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:38:21.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-29T05:38:21.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-29T05:38:21.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T05:38:21.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:38:21.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-29T05:38:22.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-29T05:38:22.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-29T05:38:22.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-29T05:38:22.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-29T05:38:22.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T05:38:22.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:38:22.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:38:22.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T05:38:22.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T05:38:00.611695+0000 2024-03-29T05:38:22.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-29T05:38:22.676 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-29T05:38:22.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-29T05:38:00.611695+0000 2024-03-29T05:38:22.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-29T05:38:22.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:38:00.611695+0000 2024-03-29T05:38:22.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:38:22.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:38:22.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:38:22.693 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T05:38:22.693 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T05:38:22.693 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:38:22.693 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:38:22.693 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T05:38:22.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:38:00.611695+0000 '>' 2024-03-29T05:38:00.611695+0000 2024-03-29T05:38:22.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:38:23.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:38:23.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:38:23.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T05:38:23.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T05:38:23.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:38:23.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:38:23.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T05:38:24.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:38:00.611695+0000 '>' 2024-03-29T05:38:00.611695+0000 2024-03-29T05:38:24.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:38:25.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:38:25.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:38:25.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T05:38:25.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T05:38:25.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:38:25.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:38:25.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T05:38:25.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:38:23.630824+0000 '>' 2024-03-29T05:38:00.611695+0000 2024-03-29T05:38:25.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:38:25.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:38:25.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:38:25.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:38:25.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:38:25.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:38:25.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:38:25.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:38:25.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:38:25.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:38:25.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:38:25.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T05:38:27.359 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-29T05:38:27.359 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T05:38:27.359 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T05:38:27.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:38:27.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:38:27.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:38:27.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:38:27.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:38:27.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:38:27.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:38:27.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:38:27.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:38:27.687 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:27.687+0000 7fdb7e3267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:27.690 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:27.691+0000 7fdb7e3267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:27.692 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:27.691+0000 7fdb7e3267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:27.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:27.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:38:28.233 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:28.231+0000 7fdb7e3267c0 -1 Falling back to public interface 2024-03-29T05:38:28.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:38:29.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:38:29.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:29.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:38:29.304 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:38:29.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:29.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:38:29.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:38:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:38:30.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:30.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:38:30.637 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:38:30.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:30.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:38:30.680 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:30.679+0000 7fdb7e3267c0 -1 osd.1 26 log_to_monitors true 2024-03-29T05:38:30.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:38:31.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:38:31.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:38:31.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:38:31.997 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:38:31.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:38:31.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:38:32.299 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/4157515238,v1:127.0.0.1:6811/4157515238] [v2:127.0.0.1:6812/4157515238,v1:127.0.0.1:6813/4157515238] exists,up 9c6b88e6-f22e-4e90-9307-358c1eee5787 2024-03-29T05:38:32.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:38:32.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:38:32.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:38:32.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:38:32.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:38:32.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:38:32.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:38:32.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:38:32.301 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:38:32.301 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:38:32.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:38:32.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:38:32.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:38:32.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:38:32.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:38:32.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:38:32.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:38:32.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:38:32.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:38:32.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:38:32.715 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T05:38:32.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:38:32.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:38:32.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:38:32.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-03-29T05:38:32.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-03-29T05:38:32.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-03-29T05:38:32.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:38:32.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:38:33.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-29T05:38:33.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-29T05:38:33.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-128849018882' 2024-03-29T05:38:33.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:38:33.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-03-29T05:38:33.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:38:33.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:38:33.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-03-29T05:38:33.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:38:33.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-03-29T05:38:33.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-03-29T05:38:33.042 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 107374182406 2024-03-29T05:38:33.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:38:33.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182406 2024-03-29T05:38:33.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:38:34.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:38:34.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:38:34.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-03-29T05:38:34.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:38:34.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-03-29T05:38:34.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:38:34.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:38:34.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-03-29T05:38:34.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:38:34.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-29T05:38:34.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-03-29T05:38:34.647 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 128849018882 2024-03-29T05:38:34.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:38:34.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-29T05:38:34.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:38:34.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:38:34.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:38:35.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T05:38:35.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:38:35.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:38:35.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:38:35.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:38:35.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:38:35.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:38:35.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:38:35.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:38:35.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:38:35.696 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:38:35.696 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:38:36.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T05:38:36.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:38:36.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:38:36.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T05:38:36.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:38:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:38:36.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:38:36.239 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:38:36.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:38:36.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:38:36.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:38:36.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:38:36.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:38:36.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:38:36.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:38:36.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:38:36.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:38:36.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:38:36.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:38:36.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:38:36.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:38:36.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:38:36.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:38:36.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:38:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:38:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:38:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:38:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:38:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:38:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:38:36.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:38:36.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:38:36.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:38:36.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:38:36.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:38:36.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:38:36.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:38:36.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:38:36.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:38:36.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:38:36.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:38:36.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:38:36.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:38:36.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:38:36.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:38:36.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:38:36.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:38:36.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:38:36.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:38:36.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:38:36.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:38:36.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:38:36.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:38:36.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:38:36.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:38:36.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:38:36.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:38:36.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:38:36.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:38:36.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:38:36.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:38:36.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:38:36.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-03-29T05:38:36.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-29T05:38:36.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-29T05:38:36.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-03-29T05:38:36.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-03-29T05:38:36.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-03-29T05:38:36.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-03-29T05:38:36.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-29T05:38:36.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-29T05:38:36.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-29T05:38:38.035 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 f81ceb70-6e5e-485f-9b09-83f09b81c74c 2024-03-29T05:38:38.303 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:38:38.360 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:38.359+0000 7fded6e567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:38.362 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:38.363+0000 7fded6e567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:38.364 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:38.363+0000 7fded6e567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:38.364 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:38.363+0000 7fded6e567c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:38:38.364 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:38.363+0000 7fded6e567c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:38:38.365 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:38.363+0000 7fded6e567c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:38:38.454 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:38.451+0000 7fded6e567c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:38:44.972 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:38:45.366 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:38:45.403 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:45.402+0000 7fe43abae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:45.434 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:45.430+0000 7fe43abae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:45.435 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:45.434+0000 7fe43abae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:38:45.645 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:38:46.028 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:46.026+0000 7fe43abae7c0 -1 Falling back to public interface 2024-03-29T05:38:46.993 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:38:48.294 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:38:48.674 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:38:48.674+0000 7fe43abae7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:38:49.605 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:38:50.996 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:38:52.356 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:38:53.803 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:38:55.133 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:38:56.607 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:38:57.927 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:38:59.246 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:39:00.570 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:39:01.892 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:39:03.207 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:39:04.923 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:39:07.034 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:39:08.942 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:39:10.452 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:39:12.341 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:39:13.902 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:39:14.225 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1729460108,v1:127.0.0.1:6803/1729460108] [v2:127.0.0.1:6804/1729460108,v1:127.0.0.1:6805/1729460108] exists,up f81ceb70-6e5e-485f-9b09-83f09b81c74c 2024-03-29T05:39:14.231 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 341c49ff-a6c2-43e2-8e41-efe08613cf56 2024-03-29T05:39:14.625 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:39:14.683 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:14.686+0000 7f3b447717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:14.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:14.730+0000 7f3b447717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:14.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:14.730+0000 7f3b447717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:14.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:14.730+0000 7f3b447717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:39:14.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:14.730+0000 7f3b447717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:39:14.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:14.730+0000 7f3b447717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:39:14.801 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:14.802+0000 7f3b447717c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:39:21.579 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:39:21.970 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:39:22.009 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:22.010+0000 7f85f3a797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:22.040 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:22.042+0000 7f85f3a797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:22.042+0000 7f85f3a797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:22.304 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:39:22.604 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:22.606+0000 7f85f3a797c0 -1 Falling back to public interface 2024-03-29T05:39:23.649 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:39:24.950 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:39:25.457 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:25.458+0000 7f85f3a797c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:39:26.290 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:39:27.777 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:39:29.098 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:39:30.579 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:39:31.898 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:39:33.219 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:39:34.713 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:39:36.032 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:39:37.426 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:39:38.729 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:39:40.049 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:39:41.779 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:39:43.447 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:39:45.314 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:39:46.666 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:39:48.565 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:39:50.358 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:39:51.817 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:39:52.137 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3668694059,v1:127.0.0.1:6811/3668694059] [v2:127.0.0.1:6812/3668694059,v1:127.0.0.1:6813/3668694059] exists,up 341c49ff-a6c2-43e2-8e41-efe08613cf56 2024-03-29T05:39:52.143 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 ebadd5d5-8cb3-4320-943f-cc1d465ab087 2024-03-29T05:39:52.544 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:39:52.606 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:52.605+0000 7f17b75357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:52.654 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:52.657+0000 7f17b75357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:52.656 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:52.657+0000 7f17b75357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:39:52.657 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:52.657+0000 7f17b75357c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:39:52.657 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:52.657+0000 7f17b75357c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:39:52.657 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:52.657+0000 7f17b75357c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:39:52.757 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:39:52.757+0000 7f17b75357c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:39:59.986 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:40:00.370 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:40:00.407 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:00.409+0000 7fcac21437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:40:00.442 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:00.441+0000 7fcac21437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:40:00.444 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:00.445+0000 7fcac21437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:40:00.715 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:40:01.009 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:01.008+0000 7fcac21437c0 -1 Falling back to public interface 2024-03-29T05:40:02.051 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:40:03.351 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:40:03.831 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:03.832+0000 7fcac21437c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:40:04.673 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:40:06.111 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:40:07.437 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:40:08.821 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:40:10.218 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:40:11.555 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:40:12.996 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:40:14.398 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:40:15.893 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:40:17.199 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:40:18.629 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:40:20.421 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:40:22.504 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:40:24.401 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:40:26.193 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:40:27.933 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:40:29.362 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:40:30.756 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:40:31.071 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2276171848,v1:127.0.0.1:6819/2276171848] [v2:127.0.0.1:6820/2276171848,v1:127.0.0.1:6821/2276171848] exists,up ebadd5d5-8cb3-4320-943f-cc1d465ab087 2024-03-29T05:40:31.516 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'testpool' created 2024-03-29T05:40:33.569 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836501 2024-03-29T05:40:35.231 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672974 2024-03-29T05:40:35.573 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509443 2024-03-29T05:40:38.306 INFO:tasks.workunit.client.0.smithi123.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-03-29T05:40:38.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-03-29T05:40:38.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-03-29T05:40:38.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-03-29T05:40:38.755 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 2 2024-03-29T05:40:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-29T05:40:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:40:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:40:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:40:38.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:40:38.779 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:40:38.779 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:40:38.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:40:38.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:40:38.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:40:38.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:40:38.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:40:38.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:40:38.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:40:38.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:40:38.898 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:40:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:40:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:40:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:40:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:40:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:40:39.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:40:39.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-29T05:40:39.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-29T05:40:39.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-29T05:40:39.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:40:39.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:40:39.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-29T05:40:39.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-29T05:40:39.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-03-29T05:40:39.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:40:39.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:40:39.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-29T05:40:39.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-29T05:40:39.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509445' 2024-03-29T05:40:39.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:40:39.809 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-29T05:40:39.809 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:40:39.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:40:39.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-29T05:40:39.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:40:39.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-29T05:40:39.812 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836504 2024-03-29T05:40:39.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-29T05:40:39.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:40:40.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-03-29T05:40:40.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:40:41.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:40:41.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:40:41.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-29T05:40:41.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:40:41.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-29T05:40:41.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:40:41.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:40:41.447 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-29T05:40:41.447 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:40:41.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-29T05:40:41.449 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672976 2024-03-29T05:40:41.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-29T05:40:41.449 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:40:41.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-29T05:40:41.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:40:41.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-29T05:40:41.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:40:41.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:40:41.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-29T05:40:41.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:40:41.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-29T05:40:41.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-29T05:40:41.783 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509445 2024-03-29T05:40:41.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:40:42.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-03-29T05:40:42.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:40:42.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:40:42.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:40:42.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:40:42.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:40:42.536 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:40:42.536 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:40:42.536 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:40:42.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:40:42.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:40:42.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:40:42.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:40:42.889 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:40:42.890 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:40:42.890 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:40:43.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:40:43.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:40:43.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:40:43.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-03-29T05:40:43.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-03-29T05:40:43.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T05:40:43.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-29T05:40:43.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:40:43.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:40:43.462 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:40:43.462 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:40:43.462 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:40:43.462 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:40:43.462 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:40:43.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:40:43.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:40:43.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:40:44.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:40:44.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:40:44.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:40:44.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:40:44.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:40:44.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:40:44.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:40:44.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:40:44.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:40:44.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:40:44.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:40:44.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:40:44.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:40:44.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:40:44.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:40:44.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:40:44.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-29T05:40:45.999 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-29T05:40:46.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T05:40:46.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:40:46.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:40:46.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:40:46.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:40:46.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:40:46.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:40:46.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:40:46.541 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:40:46.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:40:46.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:40:46.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:40:46.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:40:46.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:40:46.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:40:46.575 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:46.575+0000 7f10e209b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:40:46.578 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:46.575+0000 7f10e209b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:40:46.579 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:46.579+0000 7f10e209b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:40:46.865 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:40:46.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:40:46.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:40:47.137 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:47.135+0000 7f10e209b7c0 -1 Falling back to public interface 2024-03-29T05:40:47.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:40:48.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:40:48.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:40:48.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:40:48.205 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:40:48.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:40:48.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:40:48.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:40:49.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:40:49.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:40:49.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:40:49.527 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:40:49.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:40:49.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:40:49.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:40:50.050 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:40:50.047+0000 7f10e209b7c0 -1 osd.0 21 log_to_monitors true 2024-03-29T05:40:50.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:40:50.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:40:50.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:40:50.830 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:40:50.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:40:50.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:40:51.154 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3882776791,v1:127.0.0.1:6803/3882776791] [v2:127.0.0.1:6804/3882776791,v1:127.0.0.1:6805/3882776791] exists,up f81ceb70-6e5e-485f-9b09-83f09b81c74c 2024-03-29T05:40:51.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:40:51.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:40:51.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:40:51.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:40:51.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:40:51.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:40:51.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:40:51.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:40:51.155 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:40:51.155 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:40:51.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:40:51.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:40:51.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:40:51.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:40:51.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:40:51.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:40:51.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:40:51.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:40:51.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:40:51.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:40:51.574 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:40:51.574 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:40:51.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:40:51.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:40:51.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:40:51.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-29T05:40:51.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-29T05:40:51.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-29T05:40:51.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:40:51.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:40:51.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-29T05:40:51.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-29T05:40:51.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672979' 2024-03-29T05:40:51.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:40:51.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:40:52.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-29T05:40:52.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-29T05:40:52.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672979 2-64424509449' 2024-03-29T05:40:52.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:40:52.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-29T05:40:52.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:40:52.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:40:52.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-29T05:40:52.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:40:52.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-29T05:40:52.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-29T05:40:52.023 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 107374182402 2024-03-29T05:40:52.023 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:40:52.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-29T05:40:52.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:40:53.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:40:53.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:40:53.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-29T05:40:53.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:40:53.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-29T05:40:53.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:40:53.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:40:53.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-29T05:40:53.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:40:53.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-29T05:40:53.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-29T05:40:53.620 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672979 2024-03-29T05:40:53.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:40:53.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672979 2024-03-29T05:40:53.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:40:53.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-29T05:40:53.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:40:53.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:40:53.949 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:40:53.949 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-29T05:40:53.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-29T05:40:53.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-29T05:40:53.951 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509449 2024-03-29T05:40:53.951 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:40:54.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-29T05:40:54.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:40:54.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:40:54.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:40:54.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:40:54.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:40:54.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:40:54.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:40:54.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:40:54.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:40:54.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:40:54.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:40:54.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:40:54.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:40:54.981 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:40:54.981 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:40:55.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:40:55.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:40:55.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:40:55.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-29T05:40:55.498 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0: { 2024-03-29T05:40:55.499 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-29T05:40:55.499 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:40:55.511 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1: { 2024-03-29T05:40:55.512 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-29T05:40:55.512 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:40:55.524 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2: { 2024-03-29T05:40:55.524 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-29T05:40:55.524 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:40:55.538 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-03-29T05:40:55.538 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-29T05:40:55.538 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T05:40:55.539 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:40:55.539 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-29T05:40:55.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-03-29T05:40:55.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-03-29T05:40:55.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:40:55.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:40:55.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:40:55.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:40:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-03-29T05:40:56.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-03-29T05:40:56.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:40:56.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:40:56.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:40:56.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:40:56.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-03-29T05:40:31.446636+0000 2024-03-29T05:40:56.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-03-29T05:40:56.631 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:40:56.632 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": true, 2024-03-29T05:40:56.632 INFO:tasks.workunit.client.0.smithi123.stdout: "must": false, 2024-03-29T05:40:56.632 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "2024-03-15T05:39:16.635473+0000" 2024-03-29T05:40:56.632 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-03-29T05:40:31.446636+0000 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:40:31.446636+0000 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:40:56.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:40:56.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:40:56.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:40:56.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:40:56.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:40:31.446636+0000 '>' 2024-03-29T05:40:31.446636+0000 2024-03-29T05:40:56.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:40:57.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:40:57.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:40:57.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:40:57.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:40:57.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:40:57.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:40:57.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:40:58.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:40:31.446636+0000 '>' 2024-03-29T05:40:31.446636+0000 2024-03-29T05:40:58.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:40:59.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:40:59.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:40:59.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:40:59.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:40:59.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:40:59.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:40:59.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:40:59.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:40:56.643664+0000 '>' 2024-03-29T05:40:31.446636+0000 2024-03-29T05:40:59.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:40:59.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-29T05:40:59.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:40:59.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:40:59.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:40:59.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:40:59.564 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:40:59.564 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:40:59.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:40:59.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:40:59.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:40:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:40:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:40:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:40:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:40:59.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:40:59.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:40:59.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:40:59.988 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:40:59.989 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:40:59.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:40:59.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:40:59.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:41:00.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-03-29T05:41:00.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-03-29T05:41:00.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-03-29T05:41:00.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:00.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:41:00.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-29T05:41:00.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-29T05:41:00.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672982' 2024-03-29T05:41:00.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:00.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:41:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-29T05:41:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-29T05:41:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672982 2-64424509452' 2024-03-29T05:41:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:00.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-03-29T05:41:00.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:00.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:41:00.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-03-29T05:41:00.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:00.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-03-29T05:41:00.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-03-29T05:41:00.460 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 107374182404 2024-03-29T05:41:00.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:41:00.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182404 2024-03-29T05:41:00.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:00.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:00.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-29T05:41:00.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:41:00.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-29T05:41:00.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:00.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-29T05:41:00.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-29T05:41:00.770 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672982 2024-03-29T05:41:00.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:41:01.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-03-29T05:41:01.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:01.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-29T05:41:01.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:01.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:41:01.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-29T05:41:01.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:01.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-29T05:41:01.058 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509452 2024-03-29T05:41:01.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-29T05:41:01.058 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:41:01.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-29T05:41:01.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:41:01.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:41:01.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:41:01.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:41:01.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:41:01.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:41:01.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:41:01.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:41:01.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:41:01.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:41:01.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:41:02.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:41:02.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:41:02.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:41:02.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:41:02.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:41:02.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:41:02.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:41:02.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-03-29T05:41:02.829 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:41:02.829 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-29T05:40:56.849949+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-03-29T05:40:56.643664+0000 21'1 2024-03-29T05:40:56.643664+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:40:56.643664+0000 1 0 2024-03-29T05:41:02.829 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:41:02.829 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:41:02.829 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:41:02.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-29T05:41:02.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:41:02.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:41:02.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:41:02.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:41:02.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:41:02.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:41:02.847 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:41:03.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:41:03.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:41:03.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:41:03.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:41:03.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:41:03.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:41:03.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:41:03.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:41:03.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:41:03.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:41:03.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:41:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-29T05:41:05.401 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T05:41:05.401 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T05:41:05.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T05:41:05.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:41:05.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:41:05.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:41:05.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:41:05.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:05.692 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:05.693 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:05.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:05.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:41:05.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:41:05.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:41:05.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:41:05.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:41:05.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:41:05.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:41:05.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:41:05.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:41:05.728 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:05.727+0000 7f5756fb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:05.730 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:05.731+0000 7f5756fb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:05.732 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:05.731+0000 7f5756fb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:41:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:41:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:41:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:41:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:41:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:41:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:41:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:06.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:06.265 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:06.263+0000 7f5756fb97c0 -1 Falling back to public interface 2024-03-29T05:41:06.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:07.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:07.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:07.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:41:07.335 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:41:07.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:07.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:07.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:08.958 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:08.959+0000 7f5756fb97c0 -1 osd.0 26 log_to_monitors true 2024-03-29T05:41:08.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:09.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:09.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:09.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:41:09.977 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:41:09.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:09.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2218794801,v1:127.0.0.1:6803/2218794801] [v2:127.0.0.1:6804/2218794801,v1:127.0.0.1:6805/2218794801] exists,up f81ceb70-6e5e-485f-9b09-83f09b81c74c 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:41:10.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:41:10.299 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:41:10.299 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:41:10.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:41:10.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:41:10.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:41:10.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:41:10.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:41:10.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:41:10.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:41:10.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:41:10.401 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:41:10.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:41:10.715 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:41:10.715 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:41:10.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:41:10.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:10.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:41:10.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-29T05:41:10.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-29T05:41:10.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-29T05:41:10.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:10.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:41:11.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-03-29T05:41:11.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-03-29T05:41:11.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985' 2024-03-29T05:41:11.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:11.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:41:11.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-29T05:41:11.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-29T05:41:11.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985 2-64424509455' 2024-03-29T05:41:11.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:11.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-29T05:41:11.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:11.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:41:11.190 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-29T05:41:11.190 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:11.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-29T05:41:11.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-29T05:41:11.191 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 128849018882 2024-03-29T05:41:11.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:41:11.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-29T05:41:11.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:41:12.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:41:12.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:41:12.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-29T05:41:12.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:12.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-03-29T05:41:12.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:12.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:41:12.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-03-29T05:41:12.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:12.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-03-29T05:41:12.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-03-29T05:41:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672985 2024-03-29T05:41:12.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:41:13.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672985 2024-03-29T05:41:13.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:13.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:13.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-29T05:41:13.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:41:13.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-29T05:41:13.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:13.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-29T05:41:13.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-29T05:41:13.163 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509455 2024-03-29T05:41:13.163 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:41:13.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-29T05:41:13.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:41:13.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:41:13.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:41:13.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:41:13.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:41:13.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:41:13.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:41:13.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:41:13.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:41:13.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:41:13.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:41:14.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:41:14.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:41:14.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:41:14.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:41:14.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:41:14.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:41:14.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:41:14.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-29T05:41:14.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:41:14.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:41:14.493 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:41:14.493 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:41:14.494 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:41:14.494 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:41:14.494 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:41:14.839 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:41:14.839 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:41:14.839 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:41:15.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:41:15.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:41:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:41:17.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:41:17.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:41:17.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:41:17.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:41:17.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:41:17.434 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:41:17.434 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:17.434 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:41:17.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:41:17.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:41:17.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:41:17.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T05:41:17.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:41:17.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:41:17.436 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:41:17.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:41:17.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:41:17.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:41:17.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:41:17.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:41:17.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:41:17.469 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:17.466+0000 7fc51a8727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:17.472 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:17.470+0000 7fc51a8727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:17.474 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:17.470+0000 7fc51a8727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:41:17.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:17.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:18.018 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:18.014+0000 7fc51a8727c0 -1 Falling back to public interface 2024-03-29T05:41:18.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:19.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:19.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:19.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:41:19.066 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:41:19.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:19.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:19.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:20.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:20.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:20.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:41:20.427 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:41:20.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:20.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:20.619 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:20.618+0000 7fc51a8727c0 -1 osd.0 31 log_to_monitors true 2024-03-29T05:41:20.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:21.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:21.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:21.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:41:21.787 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:41:21.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:21.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:22.092 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/858287589,v1:127.0.0.1:6803/858287589] [v2:127.0.0.1:6804/858287589,v1:127.0.0.1:6805/858287589] exists,up f81ceb70-6e5e-485f-9b09-83f09b81c74c 2024-03-29T05:41:22.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:41:22.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:41:22.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:41:22.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:41:22.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:41:22.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:41:22.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:41:22.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:41:22.093 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:41:22.093 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:41:22.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:41:22.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:41:22.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:41:22.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:41:22.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:41:22.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:41:22.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:41:22.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:41:22.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:41:22.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:41:22.560 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:41:22.560 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:41:22.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:41:22.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:22.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:41:22.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-29T05:41:22.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-29T05:41:22.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-29T05:41:22.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:22.719 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:41:22.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-03-29T05:41:22.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-03-29T05:41:22.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672989' 2024-03-29T05:41:22.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:41:22.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:41:23.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-29T05:41:23.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-29T05:41:23.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672989 2-64424509458' 2024-03-29T05:41:23.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:23.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-29T05:41:23.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:41:23.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-29T05:41:23.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:23.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-29T05:41:23.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-29T05:41:23.031 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 150323855362 2024-03-29T05:41:23.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:41:23.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-29T05:41:23.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:41:24.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:41:24.327 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:41:24.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-29T05:41:24.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:41:25.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T05:41:25.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:41:25.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-29T05:41:25.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:25.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-03-29T05:41:25.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:25.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:41:25.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-03-29T05:41:25.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:25.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-03-29T05:41:25.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-03-29T05:41:25.964 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672989 2024-03-29T05:41:25.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:41:26.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-03-29T05:41:26.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:41:26.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-29T05:41:26.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:41:26.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:41:26.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-29T05:41:26.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:41:26.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-29T05:41:26.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-29T05:41:26.286 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509458 2024-03-29T05:41:26.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:41:26.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-29T05:41:26.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:41:26.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:41:26.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:41:27.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:41:27.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:41:27.047 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:41:27.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:41:27.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:41:27.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:41:27.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:41:27.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:41:27.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:41:27.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:41:27.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:41:27.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:41:27.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:41:27.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:41:27.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:41:27.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T05:41:27.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-29T05:41:27.724 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:40:56.639+0000 7f85d7a03640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:41:27.724 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:40:56.639+0000 7f85d7a03640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-29T05:41:27.724 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:40:56.639+0000 7f85d7a03640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-29T05:41:27.724 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:40:56.639+0000 7f85d7a03640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-29T05:41:27.724 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:40:56.639+0000 7f85d7a03640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:41:27.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:41:27.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:41:28.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:41:28.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:41:28.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:41:28.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:41:28.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:41:28.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:41:28.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:41:28.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:41:28.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:41:28.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:41:28.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:41:28.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:41:28.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:41:28.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:41:28.063 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:41:28.064 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:28.064 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:28.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:41:28.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:41:28.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:41:28.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:41:28.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:41:28.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:41:28.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:41:28.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:41:28.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:41:28.072 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:41:28.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:41:28.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:41:28.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:41:28.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:41:28.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:41:28.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:41:28.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:41:28.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:41:28.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:28.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:28.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:41:28.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:41:28.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:41:28.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:41:28.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:41:28.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:28.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:28.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-03-29T05:41:28.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-03-29T05:41:28.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:41:28.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:41:28.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:41:28.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:41:28.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:41:28.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T05:41:28.482 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:41:28.482 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:41:28.482 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:41:28.483 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:41:28.483 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:28.483 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:28.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:28.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T05:41:28.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:41:28.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:41:28.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:41:28.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:41:28.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:41:28.696 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:41:28.696 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:41:28.696 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:41:28.696 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:41:28.696 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:41:28.697 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:28.697 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:28.697 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:41:28.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:41:28.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:41:28.811 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:41:28.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:41:28.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:41:28.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:41:28.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:41:28.812 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:41:28.813 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:41:28.813 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:41:28.813 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:41:28.813 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:28.814 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:28.814 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:41:28.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:41:28.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:41:28.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-03-29T05:41:28.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:41:28.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:41:28.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:41:28.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:41:28.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:41:28.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:41:29.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:41:29.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:41:29.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:41:29.795 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:41:29.795 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:29.795 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:29.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:29.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:41:29.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:41:29.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:41:29.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:41:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:41:29.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:41:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1db1878-6f9b-44e6-8029-a116f73568bb 2024-03-29T05:41:29.842 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 e1db1878-6f9b-44e6-8029-a116f73568bb 2024-03-29T05:41:29.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e1db1878-6f9b-44e6-8029-a116f73568bb' 2024-03-29T05:41:29.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:41:29.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJVAZm/qhDMxAAdaOUyQ1gbd0BsYYIiyIGIA== 2024-03-29T05:41:29.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJVAZm/qhDMxAAdaOUyQ1gbd0BsYYIiyIGIA=="}' 2024-03-29T05:41:29.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1db1878-6f9b-44e6-8029-a116f73568bb -i td/osd-scrub-repair/0/new.json 2024-03-29T05:41:30.130 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:41:30.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:41:30.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCJVAZm/qhDMxAAdaOUyQ1gbd0BsYYIiyIGIA== --osd-uuid e1db1878-6f9b-44e6-8029-a116f73568bb 2024-03-29T05:41:30.188 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:30.190+0000 7f27d50a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:30.197 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:30.198+0000 7f27d50a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:30.198 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:30.202+0000 7f27d50a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:30.199 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:30.202+0000 7f27d50a07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:41:30.199 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:30.202+0000 7f27d50a07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:41:30.199 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:30.202+0000 7f27d50a07c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:41:30.297 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:30.298+0000 7f27d50a07c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:41:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:41:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:41:36.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:41:36.629 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:41:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:41:37.017 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:41:37.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:41:37.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:41:37.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:41:37.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:41:37.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:41:37.052 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:37.054+0000 7f64ac9c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:37.086 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:37.086+0000 7f64ac9c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:37.090+0000 7f64ac9c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:41:37.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:41:37.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:41:37.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:41:37.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:41:37.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:41:37.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:37.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:41:37.359 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:41:37.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:37.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:37.637 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:37.638+0000 7f64ac9c97c0 -1 Falling back to public interface 2024-03-29T05:41:37.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:38.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:38.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:38.676 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:41:38.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:41:38.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:38.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:38.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:39.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:39.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:39.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:41:39.996 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:41:39.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:39.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:40.278 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:41:40.278+0000 7f64ac9c97c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:41:40.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:41.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:41.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:41.310 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:41:41.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:41:41.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:41.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:41.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:42.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:42.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:42.621 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:41:42.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:41:42.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:42.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:42.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:43.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:43.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:43.959 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:41:43.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:41:43.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:43.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:44.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:45.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:45.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:45.332 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:41:45.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:41:45.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:45.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:45.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:46.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:46.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:46.654 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:41:46.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:41:46.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:46.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:46.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:47.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:47.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:47.977 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:41:47.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:41:47.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:47.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:48.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:49.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:49.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:49.360 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:41:49.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:41:49.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:49.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:49.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:50.679 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:41:50.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:41:50.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:50.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:50.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:51.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:51.994 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:41:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:41:51.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:51.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:52.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:53.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:53.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:53.370 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:41:53.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:41:53.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:53.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:53.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:54.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:54.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:54.690 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:41:54.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:41:54.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:54.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:55.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:56.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:56.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:56.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:41:56.415 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:41:56.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:56.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:56.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:57.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:57.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:57.915 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:41:57.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:41:57.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:57.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:41:58.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:41:59.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:41:59.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:41:59.602 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:41:59.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:41:59.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:41:59.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:42:00.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:01.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:01.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:01.347 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:42:01.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:42:01.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:01.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:42:02.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:03.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:03.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:03.009 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:42:03.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:42:03.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:03.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:42:03.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:04.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:04.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:04.485 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:42:04.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:42:04.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:04.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:42:04.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:05.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:05.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:05.979 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:42:05.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:42:05.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:05.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:42:06.296 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1838793790,v1:127.0.0.1:6803/1838793790] [v2:127.0.0.1:6804/1838793790,v1:127.0.0.1:6805/1838793790] exists,up e1db1878-6f9b-44e6-8029-a116f73568bb 2024-03-29T05:42:06.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:42:06.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:42:06.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:42:06.303 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:42:06.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c95cb607-738a-4e37-8baf-fa5e5034343c 2024-03-29T05:42:06.304 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 c95cb607-738a-4e37-8baf-fa5e5034343c 2024-03-29T05:42:06.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c95cb607-738a-4e37-8baf-fa5e5034343c' 2024-03-29T05:42:06.304 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:42:06.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuVAZmRi2oExAA8NUtafQKgamDxWKTnwIabA== 2024-03-29T05:42:06.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuVAZmRi2oExAA8NUtafQKgamDxWKTnwIabA=="}' 2024-03-29T05:42:06.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c95cb607-738a-4e37-8baf-fa5e5034343c -i td/osd-scrub-repair/1/new.json 2024-03-29T05:42:06.728 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:42:06.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:42:06.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCuVAZmRi2oExAA8NUtafQKgamDxWKTnwIabA== --osd-uuid c95cb607-738a-4e37-8baf-fa5e5034343c 2024-03-29T05:42:06.781 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:06.781+0000 7f945953c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:06.837 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:06.837+0000 7f945953c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:06.838 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:06.841+0000 7f945953c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:06.839 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:06.841+0000 7f945953c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:42:06.839 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:06.841+0000 7f945953c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:42:06.839 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:06.841+0000 7f945953c7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:42:06.911 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:06.913+0000 7f945953c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:42:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:42:13.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:42:13.872 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:42:13.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:42:13.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:42:14.207 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:42:14.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:42:14.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:42:14.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:42:14.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:42:14.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:42:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:14.241+0000 7f2f46d827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:14.241+0000 7f2f46d827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:14.243 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:14.245+0000 7f2f46d827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:14.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:42:14.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:42:14.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:42:14.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:42:14.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:42:14.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:14.592 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:42:14.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:42:14.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:14.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:14.820 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:14.821+0000 7f2f46d827c0 -1 Falling back to public interface 2024-03-29T05:42:14.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:15.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:15.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:15.943 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:42:15.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:42:15.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:15.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:16.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:17.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:17.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:17.260 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:42:17.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:42:17.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:17.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:17.525 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:17.525+0000 7f2f46d827c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:42:17.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:18.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:18.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:18.566 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:42:18.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:42:18.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:18.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:19.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:19.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:19.941 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:42:19.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:42:19.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:19.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:20.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:21.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:21.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:21.264 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:42:21.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:42:21.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:21.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:21.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:22.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:22.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:22.589 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:42:22.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:42:22.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:22.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:22.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:23.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:23.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:23.907 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:42:23.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:42:23.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:23.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:24.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:25.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:25.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:25.229 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:42:25.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:42:25.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:25.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:25.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:26.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:26.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:26.618 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:42:26.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:42:26.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:26.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:26.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:27.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:27.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:27.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:42:27.982 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:42:27.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:27.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:28.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:29.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:29.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:29.436 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:42:29.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:42:29.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:29.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:29.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:30.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:30.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:30.763 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:42:30.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:42:30.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:30.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:31.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:32.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:32.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:32.085 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:42:32.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:42:32.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:32.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:33.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:34.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:34.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:34.045 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:42:34.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:42:34.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:34.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:34.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:35.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:35.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:35.585 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:42:35.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:42:35.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:35.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:36.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:37.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:37.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:37.013 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:42:37.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:42:37.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:37.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:37.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:38.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:38.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:38.872 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:42:38.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:42:38.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:38.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:39.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:40.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:40.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:40.463 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:42:40.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:42:40.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:40.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:40.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:41.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:41.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:41.963 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:42:41.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:42:41.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:41.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:42.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:43.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:43.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:43.522 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:42:43.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:42:43.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:43.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:43.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:44.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:44.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:44.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T05:42:44.859 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T05:42:44.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:44.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:42:45.196 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2115974038,v1:127.0.0.1:6811/2115974038] [v2:127.0.0.1:6812/2115974038,v1:127.0.0.1:6813/2115974038] exists,up c95cb607-738a-4e37-8baf-fa5e5034343c 2024-03-29T05:42:45.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:42:45.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:42:45.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:42:45.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:42:45.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T05:42:45.205 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:42:45.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee09349f-76af-4204-9016-617e6ce0589d 2024-03-29T05:42:45.207 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 ee09349f-76af-4204-9016-617e6ce0589d 2024-03-29T05:42:45.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ee09349f-76af-4204-9016-617e6ce0589d' 2024-03-29T05:42:45.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:42:45.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVVAZmyVNaDRAA1x4T/ccWS6iLNcXbUNMEtw== 2024-03-29T05:42:45.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVVAZmyVNaDRAA1x4T/ccWS6iLNcXbUNMEtw=="}' 2024-03-29T05:42:45.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee09349f-76af-4204-9016-617e6ce0589d -i td/osd-scrub-repair/2/new.json 2024-03-29T05:42:45.585 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:42:45.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T05:42:45.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDVVAZmyVNaDRAA1x4T/ccWS6iLNcXbUNMEtw== --osd-uuid ee09349f-76af-4204-9016-617e6ce0589d 2024-03-29T05:42:45.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:45.640+0000 7fcae63fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:45.670 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:45.668+0000 7fcae63fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:45.671 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:45.672+0000 7fcae63fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:45.672 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:45.672+0000 7fcae63fc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:42:45.672 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:45.672+0000 7fcae63fc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:42:45.672 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:45.672+0000 7fcae63fc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:42:45.745 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:45.744+0000 7fcae63fc7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:42:52.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T05:42:52.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:42:52.952 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:42:52.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:42:52.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:42:53.347 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:42:53.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:42:53.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:42:53.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:42:53.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:42:53.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:42:53.382 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:53.384+0000 7f4cf00887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:53.409 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:53.408+0000 7f4cf00887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:53.410 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:53.412+0000 7f4cf00887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:42:53.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:42:53.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:42:53.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:42:53.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:42:53.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:42:53.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:53.678 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:42:53.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:42:53.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:53.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:42:53.956 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:53.956+0000 7f4cf00887c0 -1 Falling back to public interface 2024-03-29T05:42:54.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:55.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:55.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:55.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:42:55.003 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:42:55.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:55.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:42:55.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:56.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:56.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:56.312 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:42:56.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:42:56.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:56.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:42:56.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:56.687 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:42:56.688+0000 7f4cf00887c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:42:57.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:57.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:57.631 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:42:57.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:42:57.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:57.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:42:58.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:42:59.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:42:59.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:42:59.096 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:42:59.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:42:59.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:42:59.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:42:59.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:00.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:00.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:00.485 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:43:00.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:43:00.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:00.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:00.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:01.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:01.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:01.857 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:43:01.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:43:01.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:01.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:02.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:03.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:03.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:03.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:43:03.167 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:43:03.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:03.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:03.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:04.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:04.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:04.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:43:04.535 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:43:04.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:04.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:04.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:05.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:05.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:05.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:43:05.853 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:43:05.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:05.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:06.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:07.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:07.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:07.171 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:43:07.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:43:07.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:07.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:07.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:08.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:08.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:08.477 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:43:08.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:43:08.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:08.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:08.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:09.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:09.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:09.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:43:09.979 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:43:09.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:09.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:10.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:11.300 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:43:11.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:43:11.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:11.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:11.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:12.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:12.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:12.803 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:43:12.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:43:12.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:12.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:13.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:14.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:14.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:14.631 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:43:14.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:43:14.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:14.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:15.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:16.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:16.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:16.547 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:43:16.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:43:16.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:16.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:17.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:18.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:18.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:18.393 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:43:18.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:43:18.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:18.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:18.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:19.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:19.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:19.869 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:43:19.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:43:19.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:19.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:20.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:21.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:21.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:21.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:43:21.663 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:43:21.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:21.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:21.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:22.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:22.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:22.978 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:43:22.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:43:22.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:22.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:43:23.291 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2945623140,v1:127.0.0.1:6819/2945623140] [v2:127.0.0.1:6820/2945623140,v1:127.0.0.1:6821/2945623140] exists,up ee09349f-76af-4204-9016-617e6ce0589d 2024-03-29T05:43:23.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:43:23.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:43:23.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:43:23.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-03-29T05:43:23.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-29T05:43:23.718 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'testpool' created 2024-03-29T05:43:23.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:43:24.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-03-29T05:43:25.375 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 2 2024-03-29T05:43:25.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-29T05:43:25.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:43:25.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:43:25.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:43:25.397 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:43:25.398 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:43:25.398 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:43:25.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:43:25.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:43:25.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:43:25.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:43:25.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:43:25.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:43:25.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:43:25.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:43:25.515 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:43:25.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:43:25.930 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:43:25.930 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:43:25.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:43:25.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:25.930 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:43:26.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-29T05:43:26.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-29T05:43:26.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-29T05:43:26.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:26.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:43:26.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-29T05:43:26.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-29T05:43:26.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972' 2024-03-29T05:43:26.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:26.227 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:43:26.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509443 2024-03-29T05:43:26.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509443 2024-03-29T05:43:26.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972 2-64424509443' 2024-03-29T05:43:26.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:26.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-29T05:43:26.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:26.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:43:26.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-29T05:43:26.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:26.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-29T05:43:26.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-29T05:43:26.374 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836501 2024-03-29T05:43:26.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:43:26.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-29T05:43:26.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:26.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-29T05:43:26.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:26.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:43:26.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-29T05:43:26.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:26.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-29T05:43:26.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-29T05:43:26.688 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672972 2024-03-29T05:43:26.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:43:27.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-29T05:43:27.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:27.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509443 2024-03-29T05:43:27.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:27.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:43:27.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509443 2024-03-29T05:43:27.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:27.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509443 2024-03-29T05:43:27.128 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509443 2024-03-29T05:43:27.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-03-29T05:43:27.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:43:27.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509443 2024-03-29T05:43:27.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:43:28.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:43:28.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:43:28.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509443 -lt 64424509443 2024-03-29T05:43:28.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:43:28.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:28.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:29.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:43:29.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:43:29.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:43:29.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:43:29.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:43:29.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:43:29.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:43:29.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:43:29.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T05:43:29.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:43:29.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:29.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:29.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-29T05:43:29.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-29T05:43:29.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T05:43:29.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-29T05:43:29.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:43:29.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T05:43:29.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:43:29.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:43:29.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:43:30.000 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:43:30.000 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:43:30.000 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:43:30.000 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:43:30.000 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:43:30.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T05:43:30.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:43:30.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:30.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:30.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T05:43:30.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T05:43:31.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T05:43:31.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T05:43:31.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T05:43:31.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:43:31.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T05:43:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:43:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:43:31.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:43:31.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:43:31.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:43:31.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:43:31.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:43:31.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:43:31.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:43:31.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:43:31.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:31.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:31.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:43:31.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:43:31.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:43:31.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-03-29T05:43:31.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-03-29T05:43:31.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-03-29T05:43:31.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:31.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:32.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:32.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:32.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:32.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:32.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:32.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:32.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:32.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:33.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:33.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:33.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:33.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:33.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:33.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:33.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-29T05:43:33.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:43:33.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-29T05:43:33.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:43:33.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:43:33.659 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:43:33.659 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:43:33.659 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:43:33.659 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:43:33.659 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:43:33.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:43:33.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:43:33.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:43:34.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-29T05:43:34.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:43:34.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:43:34.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:43:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:43:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-29T05:43:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:43:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:43:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:43:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:43:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:43:34.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:43:34.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:43:34.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:43:34.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:43:34.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:43:34.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:43:34.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-29T05:43:34.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:43:34.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:43:34.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:43:34.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:43:34.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:43:34.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-29T05:43:36.093 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-29T05:43:36.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:43:36.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:43:36.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:43:36.629 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:43:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:43:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:43:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:43:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:43:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:43:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:43:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:43:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:43:36.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:43:36.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:43:36.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:43:36.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:43:36.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:43:36.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:43:36.667 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:36.667+0000 7f3bb36227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:43:36.670 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:36.671+0000 7f3bb36227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:43:36.671 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:36.671+0000 7f3bb36227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:36.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:37.211 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:37.211+0000 7f3bb36227c0 -1 Falling back to public interface 2024-03-29T05:43:37.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:38.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:38.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:38.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:43:38.294 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:43:38.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:38.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:38.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:39.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:39.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:39.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:43:39.612 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:43:39.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:39.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:39.751 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:39.751+0000 7f3bb36227c0 -1 osd.0 20 log_to_monitors true 2024-03-29T05:43:39.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:40.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:40.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:40.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:43:40.981 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:43:40.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:40.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:41.287 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/2771967248,v1:127.0.0.1:6803/2771967248] [v2:127.0.0.1:6804/2771967248,v1:127.0.0.1:6805/2771967248] exists,up e1db1878-6f9b-44e6-8029-a116f73568bb 2024-03-29T05:43:41.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:43:41.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:43:41.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:43:41.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:43:41.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:43:41.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:43:41.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:43:41.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:43:41.289 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:43:41.289 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:43:41.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:43:41.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:43:41.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:43:41.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:43:41.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:43:41.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:43:41.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:43:41.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:43:41.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:43:41.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:43:41.696 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:43:41.696 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:43:41.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:43:41.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:41.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:43:41.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-03-29T05:43:41.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-03-29T05:43:41.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-03-29T05:43:41.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:41.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:43:42.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-29T05:43:42.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-29T05:43:42.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672976' 2024-03-29T05:43:42.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:42.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:43:42.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-29T05:43:42.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-29T05:43:42.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672976 2-64424509447' 2024-03-29T05:43:42.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:42.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-03-29T05:43:42.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:42.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:43:42.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-03-29T05:43:42.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:42.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-03-29T05:43:42.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-03-29T05:43:42.152 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 103079215106 2024-03-29T05:43:42.152 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:43:42.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-03-29T05:43:42.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:42.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-29T05:43:42.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:42.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:43:42.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-29T05:43:42.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:42.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-29T05:43:42.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-29T05:43:42.460 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672976 2024-03-29T05:43:42.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:43:42.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-29T05:43:42.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:42.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-29T05:43:42.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:42.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:43:42.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-29T05:43:42.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:42.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-29T05:43:42.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-29T05:43:42.758 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509447 2024-03-29T05:43:42.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:43:43.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-29T05:43:43.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:43:43.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:43.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:43.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:43:43.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:43:43.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:43:43.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:43:43.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:43:43.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:43:43.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:43:43.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:43:43.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:43:43.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:43:43.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:43.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:44.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:43:44.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:43:44.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:43:44.117 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-29T05:43:44.117 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:43:44.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:43:44.118 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:43:44.118 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:43:44.118 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:43:44.119 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:43:44.119 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:43:44.447 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:43:44.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:43:44.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:43:44.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-29T05:43:44.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:43:44.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:43:44.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:43:44.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:43:44.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:43:45.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:43:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-29T05:43:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:43:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:43:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:43:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:43:45.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:43:45.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-29T05:43:46.580 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:104778fc:::obj2:head# 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:43:47.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:43:47.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:43:47.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:43:47.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:43:47.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:43:47.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:43:47.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:43:47.129 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:43:47.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:43:47.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:43:47.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:43:47.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:43:47.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:43:47.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:43:47.172 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:47.171+0000 7f390f9467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:43:47.175 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:47.175+0000 7f390f9467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:43:47.176 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:47.175+0000 7f390f9467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:43:47.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:43:47.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:43:47.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:43:47.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:43:47.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:43:47.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:47.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:43:47.442 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:43:47.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:47.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:47.712 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:47.711+0000 7f390f9467c0 -1 Falling back to public interface 2024-03-29T05:43:47.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:48.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:48.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:48.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:43:48.776 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:43:48.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:48.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:49.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:50.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:50.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:50.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:43:50.093 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:43:50.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:50.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:50.326 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:43:50.327+0000 7f390f9467c0 -1 osd.0 25 log_to_monitors true 2024-03-29T05:43:50.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:43:51.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:43:51.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:43:51.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:43:51.416 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:43:51.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:43:51.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:43:51.719 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/2256448011,v1:127.0.0.1:6803/2256448011] [v2:127.0.0.1:6804/2256448011,v1:127.0.0.1:6805/2256448011] exists,up e1db1878-6f9b-44e6-8029-a116f73568bb 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:43:51.720 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:43:51.721 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:43:51.721 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:43:51.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:43:51.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:43:51.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:43:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:43:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:43:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:43:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:43:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:43:51.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:43:52.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:43:52.257 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:43:52.257 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:43:52.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:43:52.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:52.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:43:52.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-29T05:43:52.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-29T05:43:52.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-29T05:43:52.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:52.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:43:52.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-29T05:43:52.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-29T05:43:52.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672979' 2024-03-29T05:43:52.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:43:52.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:43:52.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-29T05:43:52.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-29T05:43:52.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672979 2-64424509450' 2024-03-29T05:43:52.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:52.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-29T05:43:52.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:52.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:43:52.703 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-29T05:43:52.703 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:52.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-29T05:43:52.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-29T05:43:52.705 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 124554051586 2024-03-29T05:43:52.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:43:53.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-29T05:43:53.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:43:54.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:43:54.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:43:54.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-29T05:43:54.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:43:55.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T05:43:55.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:43:55.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-29T05:43:55.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:55.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-29T05:43:55.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:55.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:43:55.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-29T05:43:55.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:55.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-29T05:43:55.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-29T05:43:55.671 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672979 2024-03-29T05:43:55.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:43:56.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672979 2024-03-29T05:43:56.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:43:56.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-29T05:43:56.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:43:56.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:43:56.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-29T05:43:56.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:43:56.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-29T05:43:56.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-29T05:43:56.042 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509450 2024-03-29T05:43:56.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:43:56.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509450 2024-03-29T05:43:56.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:43:56.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:56.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:56.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:43:56.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:43:56.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:43:56.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:43:56.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:43:56.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:43:56.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:43:56.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:43:57.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:43:57.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:43:57.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:43:57.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:43:57.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:43:57.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:43:57.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:43:57.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-29T05:43:57.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:43:57.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:43:57.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:43:57.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:43:57.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:43:57.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:43:57.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:43:57.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:43:57.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:43:57.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:43:57.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:43:57.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:43:58.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:44:00.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:44:00.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:44:00.347 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:44:00.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:44:00.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:44:00.349 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:44:00.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:44:00.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:44:00.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:44:00.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:44:00.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:44:00.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:44:00.385 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:00.383+0000 7feb32b587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:00.406 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:00.407+0000 7feb32b587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:00.407 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:00.407+0000 7feb32b587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:44:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:44:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:44:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:44:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:44:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:44:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:44:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:00.948 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:00.947+0000 7feb32b587c0 -1 Falling back to public interface 2024-03-29T05:44:01.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:02.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:02.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:02.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:44:02.019 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:02.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:02.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:02.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:03.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:03.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:03.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:44:03.339 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:44:03.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:03.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:03.644 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:03.643+0000 7feb32b587c0 -1 osd.1 30 log_to_monitors true 2024-03-29T05:44:03.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:04.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:04.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:04.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:44:04.663 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:44:04.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:04.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/3042865113,v1:127.0.0.1:6811/3042865113] [v2:127.0.0.1:6812/3042865113,v1:127.0.0.1:6813/3042865113] exists,up c95cb607-738a-4e37-8baf-fa5e5034343c 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:44:05.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:44:05.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:44:05.002 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:44:05.002 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:44:05.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:44:05.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:44:05.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:44:05.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:44:05.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:44:05.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:44:05.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:44:05.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:44:05.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:44:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:44:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:44:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:44:05.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:05.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:44:05.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-29T05:44:05.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-29T05:44:05.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-03-29T05:44:05.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:05.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:44:05.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-03-29T05:44:05.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-03-29T05:44:05.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-146028888066' 2024-03-29T05:44:05.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:05.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:44:05.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-29T05:44:05.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-29T05:44:05.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-146028888066 2-64424509454' 2024-03-29T05:44:05.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:05.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-03-29T05:44:05.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:05.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:44:05.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-03-29T05:44:05.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:05.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-29T05:44:05.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-03-29T05:44:05.900 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 124554051589 2024-03-29T05:44:05.900 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:06.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-03-29T05:44:06.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:44:07.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:44:07.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:07.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051589 2024-03-29T05:44:07.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:07.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:07.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888066 2024-03-29T05:44:07.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:44:07.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888066 2024-03-29T05:44:07.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:07.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-03-29T05:44:07.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2024-03-29T05:44:07.526 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 146028888066 2024-03-29T05:44:07.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:44:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-03-29T05:44:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:07.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:07.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-29T05:44:07.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:44:07.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-29T05:44:07.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-29T05:44:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-29T05:44:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509454 2024-03-29T05:44:07.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:44:08.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-29T05:44:08.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:44:08.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:08.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:08.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:44:08.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:44:08.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:44:08.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:44:08.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:44:08.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:44:08.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:44:08.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:44:08.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:44:08.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:44:08.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:08.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:09.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:44:09.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:44:09.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:44:09.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-03-29T05:44:09.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-29T05:44:09.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-29T05:44:09.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-29T05:44:09.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:44:09.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-03-29T05:44:09.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-03-29T05:44:09.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:44:09.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-29T05:44:09.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:44:09.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-29T05:44:09.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-03-29T05:44:09.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-03-29T05:44:09.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:09.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:09.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:09.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:10.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-03-29T05:43:23.654394+0000 2024-03-29T05:44:10.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-03-29T05:44:10.410 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:44:10.410 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": true, 2024-03-29T05:44:10.410 INFO:tasks.workunit.client.0.smithi123.stdout: "must": false, 2024-03-29T05:44:10.410 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "2024-03-15T05:42:30.415406+0000" 2024-03-29T05:44:10.410 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:44:10.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-03-29T05:43:23.654394+0000 2024-03-29T05:44:10.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:43:23.654394+0000 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:10.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:10.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:43:23.654394+0000 '>' 2024-03-29T05:43:23.654394+0000 2024-03-29T05:44:10.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:44:11.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:44:11.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:11.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:11.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:11.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:11.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:11.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:12.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:43:23.654394+0000 '>' 2024-03-29T05:43:23.654394+0000 2024-03-29T05:44:12.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:44:13.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:44:13.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:13.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:13.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:13.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:13.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:13.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:13.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:43:23.654394+0000 '>' 2024-03-29T05:43:23.654394+0000 2024-03-29T05:44:13.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:44:14.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:44:14.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:14.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:14.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:14.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:14.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:14.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:14.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:44:11.258766+0000 '>' 2024-03-29T05:43:23.654394+0000 2024-03-29T05:44:14.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:44:14.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-29T05:44:14.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:44:14.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:44:14.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:44:14.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:44:14.665 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:44:14.665 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:44:14.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:44:14.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:44:14.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:44:14.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:44:14.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:44:14.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:44:14.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:44:14.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:44:14.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:44:15.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:44:15.080 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:15.080 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:44:15.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:44:15.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:15.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:44:15.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051592 2024-03-29T05:44:15.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051592 2024-03-29T05:44:15.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051592' 2024-03-29T05:44:15.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:15.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:44:15.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-03-29T05:44:15.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-03-29T05:44:15.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051592 1-146028888069' 2024-03-29T05:44:15.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:15.358 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:44:15.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-29T05:44:15.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-29T05:44:15.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051592 1-146028888069 2-64424509457' 2024-03-29T05:44:15.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:15.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051592 2024-03-29T05:44:15.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:15.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:44:15.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051592 2024-03-29T05:44:15.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:15.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051592 2024-03-29T05:44:15.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2024-03-29T05:44:15.492 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 124554051592 2024-03-29T05:44:15.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:15.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051592 2024-03-29T05:44:15.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:44:16.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:44:16.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:17.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051593 -lt 124554051592 2024-03-29T05:44:17.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:17.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888069 2024-03-29T05:44:17.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:17.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:44:17.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888069 2024-03-29T05:44:17.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:17.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-03-29T05:44:17.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2024-03-29T05:44:17.149 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 146028888069 2024-03-29T05:44:17.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:44:17.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888069 -lt 146028888069 2024-03-29T05:44:17.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:17.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-29T05:44:17.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:17.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:44:17.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:17.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-29T05:44:17.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-29T05:44:17.485 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509457 2024-03-29T05:44:17.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-29T05:44:17.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:44:17.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-29T05:44:17.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:44:17.808 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:17.808 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:18.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:44:18.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:44:18.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:44:18.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:44:18.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:44:18.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:44:18.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:44:18.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:44:18.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:44:18.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:44:18.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:18.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:18.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:44:18.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:44:18.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:44:18.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-29T05:44:18.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:44:18.905 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:44:19.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:44:19.202 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:19.203 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:44:19.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:44:19.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:19.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:44:19.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051594 2024-03-29T05:44:19.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051594 2024-03-29T05:44:19.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594' 2024-03-29T05:44:19.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:19.377 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:44:19.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888071 2024-03-29T05:44:19.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888071 2024-03-29T05:44:19.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594 1-146028888071' 2024-03-29T05:44:19.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:19.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:44:19.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-29T05:44:19.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-29T05:44:19.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594 1-146028888071 2-64424509459' 2024-03-29T05:44:19.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:19.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051594 2024-03-29T05:44:19.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:19.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:44:19.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051594 2024-03-29T05:44:19.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:19.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051594 2024-03-29T05:44:19.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-03-29T05:44:19.661 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 124554051594 2024-03-29T05:44:19.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:19.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051593 -lt 124554051594 2024-03-29T05:44:19.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:44:20.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:44:20.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:21.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051594 -lt 124554051594 2024-03-29T05:44:21.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:21.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888071 2024-03-29T05:44:21.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:21.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:44:21.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888071 2024-03-29T05:44:21.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:21.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888071 2024-03-29T05:44:21.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888071' 2024-03-29T05:44:21.272 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 146028888071 2024-03-29T05:44:21.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:44:21.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888071 -lt 146028888071 2024-03-29T05:44:21.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:21.590 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:21.590 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-29T05:44:21.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:44:21.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-29T05:44:21.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:21.593 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509459 2024-03-29T05:44:21.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-29T05:44:21.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-29T05:44:21.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:44:21.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-29T05:44:21.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-29T05:44:21.922 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:44:21.922 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-29T05:44:21.922 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-29T05:44:21.923 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-29T05:44:21.923 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:44:21.923 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.251+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:44:21.923 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.251+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-29T05:44:21.923 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.255+0000 7feb16ae2640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:44:21.923 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.255+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-29T05:44:21.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-03-29T05:44:21.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-29T05:44:22.233 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:44:22.233 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-29T05:44:11.259009+0000 20'10 35:100 [1,0] 1 [1,0] 1 20'10 2024-03-29T05:44:11.258766+0000 20'10 2024-03-29T05:44:11.258766+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:44:11.258766+0000 10 0 2024-03-29T05:44:22.233 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:44:22.233 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:44:22.233 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:44:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-03-29T05:44:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-29T05:44:22.538 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:44:22.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-03-29T05:44:22.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:44:22.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-29T05:44:22.555 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-29T05:44:22.555 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:44:22.555 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-29T05:44:22.556 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-29T05:44:22.556 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:44:22.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:44:22.869 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-29T05:44:22.869 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:44:23.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:44:23.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:44:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-03-29T05:44:25.185 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T05:44:25.185 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:44:25.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:44:25.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:44:25.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:44:25.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:44:25.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:44:25.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:44:25.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:44:25.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:44:25.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:44:25.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:44:25.474 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:44:25.474 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:44:25.474 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:44:25.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:44:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:44:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:44:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:44:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:44:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:44:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:44:25.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:44:25.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:44:25.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:44:25.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:44:25.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:44:25.512 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:25.511+0000 7fb07a6577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:25.514 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:25.515+0000 7fb07a6577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:25.515 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:25.515+0000 7fb07a6577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:25.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:44:26.052 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:26.051+0000 7fb07a6577c0 -1 Falling back to public interface 2024-03-29T05:44:26.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:27.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:27.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:27.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:44:27.138 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:27.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:27.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:44:27.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:28.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:28.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:28.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:44:28.457 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:44:28.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:28.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:44:28.755 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:28.755+0000 7fb07a6577c0 -1 osd.0 35 log_to_monitors true 2024-03-29T05:44:28.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:29.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:29.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:29.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:44:29.786 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:44:29.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:29.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:44:30.127 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 39 up_thru 31 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6802/3547961455,v1:127.0.0.1:6803/3547961455] [v2:127.0.0.1:6804/3547961455,v1:127.0.0.1:6805/3547961455] exists,up e1db1878-6f9b-44e6-8029-a116f73568bb 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:44:30.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:44:30.129 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:44:30.129 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:44:30.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:44:30.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:44:30.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:44:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:44:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:44:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:44:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:44:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:44:30.229 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:44:30.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:44:30.578 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:30.578 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:44:30.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:44:30.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:30.578 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:44:30.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-29T05:44:30.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-29T05:44:30.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546' 2024-03-29T05:44:30.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:30.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:44:30.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888074 2024-03-29T05:44:30.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888074 2024-03-29T05:44:30.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546 1-146028888074' 2024-03-29T05:44:30.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:30.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:44:31.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-29T05:44:31.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-29T05:44:31.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546 1-146028888074 2-64424509462' 2024-03-29T05:44:31.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:31.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724546 2024-03-29T05:44:31.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:31.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:44:31.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724546 2024-03-29T05:44:31.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:31.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-29T05:44:31.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2024-03-29T05:44:31.058 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 167503724546 2024-03-29T05:44:31.058 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:31.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 167503724546 2024-03-29T05:44:31.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:44:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:44:32.359 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:32.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-29T05:44:32.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:32.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:32.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888074 2024-03-29T05:44:32.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:44:32.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888074 2024-03-29T05:44:32.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:32.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888074 2024-03-29T05:44:32.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888074' 2024-03-29T05:44:32.679 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 146028888074 2024-03-29T05:44:32.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:44:33.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888074 -lt 146028888074 2024-03-29T05:44:33.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:33.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-29T05:44:33.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:33.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:44:33.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-29T05:44:33.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:33.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-29T05:44:33.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-29T05:44:33.038 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509462 2024-03-29T05:44:33.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:44:33.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-29T05:44:33.363 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:44:33.363 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:33.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:33.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:44:33.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:44:33.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:44:33.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:44:33.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:44:33.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:44:33.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:44:33.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:44:34.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:44:34.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:44:34.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:34.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:34.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:44:34.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:44:34.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:44:34.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-03-29T05:44:34.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T05:44:34.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-29T05:44:34.475 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:44:34.475 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-29T05:44:34.476 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-29T05:44:34.476 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-29T05:44:34.476 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:44:34.476 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.251+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:44:34.476 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.251+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-29T05:44:34.476 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.255+0000 7feb16ae2640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:44:34.476 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.255+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-29T05:44:34.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-29T05:44:34.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:44:34.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:44:34.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:44:34.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:44:34.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:44:34.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:44:35.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-03-29T05:44:36.540 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-03-29T05:44:36.540 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:104778fc:::obj2:head# 2024-03-29T05:44:37.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:44:37.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:44:37.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:44:37.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:44:37.078 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:44:37.078 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:44:37.078 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:44:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:44:37.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:44:37.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:44:37.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:44:37.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:44:37.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:44:37.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:44:37.121 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:37.118+0000 7fd1ed4bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:37.123 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:37.122+0000 7fd1ed4bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:37.124 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:37.122+0000 7fd1ed4bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:44:37.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:37.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:37.659 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:37.658+0000 7fd1ed4bf7c0 -1 Falling back to public interface 2024-03-29T05:44:37.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:38.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:38.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:38.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:44:38.757 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:38.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:39.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:40.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:40.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:40.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:44:40.075 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:44:40.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:40.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:40.186 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:44:40.186+0000 7fd1ed4bf7c0 -1 osd.1 40 log_to_monitors true 2024-03-29T05:44:40.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:44:41.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:44:41.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:44:41.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:44:41.423 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:44:41.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:44:41.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:44:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 41 last_clean_interval [34,40) [v2:127.0.0.1:6810/1681044363,v1:127.0.0.1:6811/1681044363] [v2:127.0.0.1:6812/1681044363,v1:127.0.0.1:6813/1681044363] exists,up c95cb607-738a-4e37-8baf-fa5e5034343c 2024-03-29T05:44:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:44:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:44:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:44:41.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:44:41.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:44:41.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:44:41.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:44:41.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:44:41.748 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:44:41.748 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:44:41.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:44:41.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:44:41.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:44:41.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:44:41.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:44:41.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:44:41.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:44:41.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:44:41.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:44:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:44:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:44:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:44:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:42.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:44:42.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724549 2024-03-29T05:44:42.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724549 2024-03-29T05:44:42.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724549' 2024-03-29T05:44:42.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:42.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:44:42.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561026 2024-03-29T05:44:42.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561026 2024-03-29T05:44:42.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724549 1-188978561026' 2024-03-29T05:44:42.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:42.498 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:44:42.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-29T05:44:42.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-29T05:44:42.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724549 1-188978561026 2-64424509465' 2024-03-29T05:44:42.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:42.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724549 2024-03-29T05:44:42.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:42.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:44:42.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724549 2024-03-29T05:44:42.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:42.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724549 2024-03-29T05:44:42.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724549' 2024-03-29T05:44:42.640 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 167503724549 2024-03-29T05:44:42.641 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:42.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724548 -lt 167503724549 2024-03-29T05:44:42.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:44:43.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:44:43.990 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:44.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724548 -lt 167503724549 2024-03-29T05:44:44.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:44:45.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T05:44:45.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:45.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724550 -lt 167503724549 2024-03-29T05:44:45.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:45.629 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561026 2024-03-29T05:44:45.629 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:45.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:44:45.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561026 2024-03-29T05:44:45.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:45.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561026 2024-03-29T05:44:45.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561026' 2024-03-29T05:44:45.633 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 188978561026 2024-03-29T05:44:45.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:44:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561026 -lt 188978561026 2024-03-29T05:44:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:45.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-29T05:44:45.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:45.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:44:45.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-29T05:44:45.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:45.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-29T05:44:45.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-29T05:44:45.972 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509465 2024-03-29T05:44:45.973 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:44:46.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509466 -lt 64424509465 2024-03-29T05:44:46.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:44:46.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:46.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:44:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:44:46.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:44:46.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:44:46.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:44:46.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:44:46.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:44:46.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:44:47.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:44:47.086 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:44:47.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:44:47.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:47.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:47.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:47.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:47.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:47.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-29T05:44:48.080 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-29T05:44:48.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:48.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-29T05:44:48.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:48.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:44:48.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:44:48.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:48.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:48.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:48.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:48.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:48.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:48.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:44:11.258766+0000 '>' 2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:48.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:44:49.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:44:49.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:49.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:49.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:49.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:49.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:49.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:49.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:44:11.258766+0000 '>' 2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:49.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:44:50.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:44:50.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:50.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:50.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:50.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:50.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:50.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:51.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:44:11.258766+0000 '>' 2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:51.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:44:52.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:44:52.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:52.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:52.034 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:52.034 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:52.034 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:52.034 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:52.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:44:11.258766+0000 '>' 2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:52.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:44:53.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:44:53.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:44:53.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:44:53.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:44:53.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:44:53.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:44:53.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:44:53.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:44:48.374880+0000 '>' 2024-03-29T05:44:11.258766+0000 2024-03-29T05:44:53.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:44:53.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-03-29T05:44:55.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-29T05:44:55.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:44:55.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:44:55.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:44:55.954 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:44:55.954 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:44:55.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:44:55.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:55.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:44:56.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724553 2024-03-29T05:44:56.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724553 2024-03-29T05:44:56.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724553' 2024-03-29T05:44:56.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:56.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:44:56.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561030 2024-03-29T05:44:56.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561030 2024-03-29T05:44:56.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724553 1-188978561030' 2024-03-29T05:44:56.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:44:56.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:44:56.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509469 2024-03-29T05:44:56.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509469 2024-03-29T05:44:56.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724553 1-188978561030 2-64424509469' 2024-03-29T05:44:56.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:56.403 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724553 2024-03-29T05:44:56.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:56.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:44:56.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724553 2024-03-29T05:44:56.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:56.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724553 2024-03-29T05:44:56.407 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 167503724553 2024-03-29T05:44:56.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724553' 2024-03-29T05:44:56.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:44:56.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724553 -lt 167503724553 2024-03-29T05:44:56.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:56.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561030 2024-03-29T05:44:56.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:56.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:44:56.703 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561030 2024-03-29T05:44:56.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:56.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561030 2024-03-29T05:44:56.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561030' 2024-03-29T05:44:56.705 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 188978561030 2024-03-29T05:44:56.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:44:57.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561030 -lt 188978561030 2024-03-29T05:44:57.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:44:57.050 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:44:57.050 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509469 2024-03-29T05:44:57.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:44:57.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509469 2024-03-29T05:44:57.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:44:57.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509469 2024-03-29T05:44:57.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-03-29T05:44:57.055 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509469 2024-03-29T05:44:57.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:44:57.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509469 2024-03-29T05:44:57.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:44:58.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:44:58.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:44:58.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509469 2024-03-29T05:44:58.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-29T05:44:58.990 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:44:58.990 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-03-29T05:44:48.375028+0000 20'10 45:138 [1,0] 1 [1,0] 1 20'10 2024-03-29T05:44:48.374880+0000 20'10 2024-03-29T05:44:48.374880+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:44:48.374880+0000 9 0 2024-03-29T05:44:58.990 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:44:58.990 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:44:58.990 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:44:59.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-29T05:44:59.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-03-29T05:44:59.298 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:44:59.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-29T05:44:59.318 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:10.795+0000 7feb16ae2640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.251+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.251+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.255+0000 7feb16ae2640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:44:59.319 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:11.255+0000 7feb16ae2640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=17/17 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-29T05:44:59.320 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:48.374+0000 7fd1d1449640 10 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=10 ec=17/17 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-03-29T05:44:59.320 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:48.374+0000 7fd1d1449640 10 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=10 ec=17/17 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-29T05:44:59.320 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:48.374+0000 7fd1d1449640 15 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=9 ec=17/17 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-29T05:44:59.320 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:48.374+0000 7fd1d1449640 20 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=9 ec=17/17 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-03-29T05:44:59.320 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:44:48.374+0000 7fd1d1449640 19 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=9 ec=17/17 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:44:59.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:44:59.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:44:59.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:44:59.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:44:59.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:44:59.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:44:59.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:44:59.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:44:59.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:44:59.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:44:59.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:44:59.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:44:59.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:44:59.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:44:59.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:44:59.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:44:59.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:44:59.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:44:59.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:44:59.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:44:59.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:44:59.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:44:59.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:44:59.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:44:59.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:44:59.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:44:59.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:44:59.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:44:59.464 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:44:59.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:44:59.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:44:59.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:44:59.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:44:59.465 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:44:59.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:44:59.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:44:59.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:44:59.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:44:59.467 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:44:59.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:44:59.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:44:59.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:44:59.469 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:44:59.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:44:59.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:44:59.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:44:59.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:44:59.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:44:59.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:44:59.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:44:59.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:44:59.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:44:59.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:44:59.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:44:59.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:44:59.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:44:59.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:44:59.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:44:59.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:44:59.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:44:59.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:44:59.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:44:59.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T05:44:59.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:44:59.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:44:59.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:44:59.860 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:44:59.860 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:44:59.860 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:44:59.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:44:59.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T05:45:00.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:45:00.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:45:00.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:45:00.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:45:00.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:45:00.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:45:00.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:45:00.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:45:00.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:45:00.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:45:00.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:45:00.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:45:00.175 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:45:00.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:45:00.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:45:00.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:45:00.176 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:45:00.176 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:45:00.176 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:45:00.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:45:00.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:45:00.177 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:45:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-03-29T05:45:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:45:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:45:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:45:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:45:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:45:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:45:01.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:45:01.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:45:01.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:45:01.162 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:45:01.162 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:45:01.162 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:45:01.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:45:01.163 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:45:01.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:45:01.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:45:01.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-03-29T05:45:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-29T05:45:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:45:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:45:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:45:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:45:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:45:01.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:45:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:45:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:45:01.205 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:45:01.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e8149d32-5e6c-42c1-99df-4bf2d0d4fde3 2024-03-29T05:45:01.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e8149d32-5e6c-42c1-99df-4bf2d0d4fde3' 2024-03-29T05:45:01.205 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 e8149d32-5e6c-42c1-99df-4bf2d0d4fde3 2024-03-29T05:45:01.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:45:01.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdVQZm23OBDRAArgraAmaUhviPd7wYyeXcIw== 2024-03-29T05:45:01.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdVQZm23OBDRAArgraAmaUhviPd7wYyeXcIw=="}' 2024-03-29T05:45:01.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e8149d32-5e6c-42c1-99df-4bf2d0d4fde3 -i td/osd-scrub-repair/0/new.json 2024-03-29T05:45:01.457 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:45:01.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:45:01.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBdVQZm23OBDRAArgraAmaUhviPd7wYyeXcIw== --osd-uuid e8149d32-5e6c-42c1-99df-4bf2d0d4fde3 2024-03-29T05:45:01.507 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:01.510+0000 7f7838dbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:01.509 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:01.514+0000 7f7838dbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:01.510 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:01.514+0000 7f7838dbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:01.511 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:01.514+0000 7f7838dbc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:45:01.511 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:01.514+0000 7f7838dbc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:45:01.511 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:01.514+0000 7f7838dbc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:45:01.582 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:01.586+0000 7f7838dbc7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:45:07.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:45:07.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:45:07.544 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:45:07.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:45:07.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:45:07.954 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:45:07.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:45:07.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:45:07.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:45:07.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:45:07.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:45:07.989 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:07.994+0000 7f5655c077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:07.991 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:07.994+0000 7f5655c077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:07.993 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:07.994+0000 7f5655c077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:08.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:45:08.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:45:08.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:45:08.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:45:08.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:45:08.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:08.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:45:08.295 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:45:08.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:08.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:08.526 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:08.530+0000 7f5655c077c0 -1 Falling back to public interface 2024-03-29T05:45:08.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:09.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:09.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:09.628 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:45:09.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:45:09.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:09.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:09.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:10.808 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:10.810+0000 7f5655c077c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:45:10.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:10.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:10.947 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:45:10.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:45:10.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:10.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:11.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:12.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:12.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:12.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:45:12.375 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:45:12.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:12.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:12.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:13.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:13.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:13.692 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:45:13.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:45:13.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:13.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:13.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:14.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:14.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:14.994 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:45:14.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:45:14.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:14.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:15.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:16.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:16.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:16.316 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:45:16.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:45:16.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:16.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:16.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:17.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:17.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:17.722 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:45:17.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:45:17.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:17.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:18.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:19.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:19.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:19.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:45:19.037 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:45:19.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:19.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:19.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:20.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:20.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:20.352 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:45:20.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:45:20.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:20.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:20.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:21.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:21.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:21.684 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:45:21.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:45:21.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:21.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:22.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:23.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:23.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:23.002 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:45:23.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:45:23.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:23.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:23.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:24.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:24.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:24.593 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:45:24.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:45:24.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:24.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:24.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:25.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:25.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:25.986 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:45:25.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:45:25.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:25.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:26.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:27.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:27.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:27.733 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:45:27.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:45:27.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:27.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:28.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:29.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:29.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:29.570 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:45:29.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:45:29.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:29.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:30.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:31.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:31.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:31.174 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:45:31.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:45:31.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:31.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:31.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:32.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:32.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:32.519 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:45:32.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:45:32.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:32.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:45:32.832 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3272661410,v1:127.0.0.1:6803/3272661410] [v2:127.0.0.1:6804/3272661410,v1:127.0.0.1:6805/3272661410] exists,up e8149d32-5e6c-42c1-99df-4bf2d0d4fde3 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:45:32.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:45:32.836 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:45:32.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:45:32.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:45:32.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:45:32.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:45:32.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=625fccf2-1a94-43cb-b66b-4b3c935de44d 2024-03-29T05:45:32.838 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 625fccf2-1a94-43cb-b66b-4b3c935de44d 2024-03-29T05:45:32.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 625fccf2-1a94-43cb-b66b-4b3c935de44d' 2024-03-29T05:45:32.839 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:45:32.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8VQZmAyaHMxAAMXYwHijfRT177FmHfQ9bIQ== 2024-03-29T05:45:32.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8VQZmAyaHMxAAMXYwHijfRT177FmHfQ9bIQ=="}' 2024-03-29T05:45:32.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 625fccf2-1a94-43cb-b66b-4b3c935de44d -i td/osd-scrub-repair/1/new.json 2024-03-29T05:45:33.217 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:45:33.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:45:33.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB8VQZmAyaHMxAAMXYwHijfRT177FmHfQ9bIQ== --osd-uuid 625fccf2-1a94-43cb-b66b-4b3c935de44d 2024-03-29T05:45:33.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:33.273+0000 7fdf4d99a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:33.293 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:33.297+0000 7fdf4d99a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:33.294 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:33.297+0000 7fdf4d99a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:33.295 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:33.297+0000 7fdf4d99a7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:45:33.295 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:33.297+0000 7fdf4d99a7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:45:33.295 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:33.297+0000 7fdf4d99a7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:45:33.361 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:33.365+0000 7fdf4d99a7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:45:39.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:45:39.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:45:39.236 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:45:39.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:45:39.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:45:39.624 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:45:39.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:45:39.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:45:39.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:45:39.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:45:39.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:45:39.661 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:39.665+0000 7f54f42a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:39.680 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:39.681+0000 7f54f42a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:39.681 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:39.685+0000 7f54f42a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:45:39.935 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:45:39.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:39.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:40.222 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:40.225+0000 7f54f42a87c0 -1 Falling back to public interface 2024-03-29T05:45:40.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:41.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:41.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:41.228 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:45:41.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:45:41.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:41.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:41.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:42.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:42.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:42.550 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:45:42.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:45:42.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:42.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:42.823 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:45:42.825+0000 7f54f42a87c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:45:42.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:43.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:43.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:43.853 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:45:43.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:45:43.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:43.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:44.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:45.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:45.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:45.170 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:45:45.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:45:45.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:45.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:45.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:46.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:46.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:46.480 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:45:46.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:45:46.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:46.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:46.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:47.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:47.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:47.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:45:47.898 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:45:47.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:47.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:48.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:49.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:49.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:49.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:45:49.222 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:45:49.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:49.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:49.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:50.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:50.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:50.609 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:45:50.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:45:50.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:50.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:51.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:52.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:52.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:52.026 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:45:52.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:45:52.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:52.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:52.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:53.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:53.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:53.341 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:45:53.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:45:53.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:53.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:53.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:54.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:54.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:54.748 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:45:54.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:45:54.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:54.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:55.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:56.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:56.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:56.513 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:45:56.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:45:56.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:56.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:57.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:45:58.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:45:58.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:45:58.212 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:45:58.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:45:58.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:45:58.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:45:59.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:00.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:00.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:00.243 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:46:00.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:46:00.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:00.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:46:00.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:01.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:01.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:01.835 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:46:01.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:46:01.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:01.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:46:02.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:03.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:03.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:03.651 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:46:03.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:46:03.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:03.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:46:04.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:05.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:05.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:05.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:46:05.007 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:46:05.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:05.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:46:05.347 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3930864240,v1:127.0.0.1:6811/3930864240] [v2:127.0.0.1:6812/3930864240,v1:127.0.0.1:6813/3930864240] exists,up 625fccf2-1a94-43cb-b66b-4b3c935de44d 2024-03-29T05:46:05.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:46:05.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:46:05.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:46:05.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:46:05.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T05:46:05.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:46:05.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd6a5d1f-3a4d-49e3-8791-f78ea09d9756 2024-03-29T05:46:05.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bd6a5d1f-3a4d-49e3-8791-f78ea09d9756' 2024-03-29T05:46:05.354 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 bd6a5d1f-3a4d-49e3-8791-f78ea09d9756 2024-03-29T05:46:05.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:46:05.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdVQZm6/q5FhAAL7egPdVgR9GLY6EN/Ut8Kg== 2024-03-29T05:46:05.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdVQZm6/q5FhAAL7egPdVgR9GLY6EN/Ut8Kg=="}' 2024-03-29T05:46:05.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd6a5d1f-3a4d-49e3-8791-f78ea09d9756 -i td/osd-scrub-repair/2/new.json 2024-03-29T05:46:05.732 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:46:05.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T05:46:05.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCdVQZm6/q5FhAAL7egPdVgR9GLY6EN/Ut8Kg== --osd-uuid bd6a5d1f-3a4d-49e3-8791-f78ea09d9756 2024-03-29T05:46:05.787 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:05.789+0000 7fa44d1947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:05.819 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:05.821+0000 7fa44d1947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:05.820 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:05.825+0000 7fa44d1947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:05.825+0000 7fa44d1947c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:46:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:05.825+0000 7fa44d1947c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:46:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:05.825+0000 7fa44d1947c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:46:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:05.873+0000 7fa44d1947c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:46:11.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T05:46:11.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:46:11.843 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:46:11.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:46:11.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:46:12.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:46:12.234 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:46:12.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:46:12.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:46:12.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:46:12.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:46:12.268 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:12.273+0000 7f4ece6fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:12.289 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:12.293+0000 7f4ece6fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:12.290 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:12.293+0000 7f4ece6fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:12.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:46:12.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:46:12.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:46:12.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:46:12.550 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:46:12.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:46:12.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:12.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:46:12.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:12.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:12.858 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:12.861+0000 7f4ece6fe7c0 -1 Falling back to public interface 2024-03-29T05:46:12.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:13.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:13.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:13.885 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:46:13.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:46:13.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:13.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:14.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:15.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:15.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:15.191 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:46:15.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:46:15.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:15.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:15.330 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:15.333+0000 7f4ece6fe7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:46:15.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:16.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:16.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:16.531 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:46:16.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:46:16.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:16.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:16.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:17.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:17.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:17.942 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:46:17.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:46:17.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:17.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:18.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:19.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:19.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:19.372 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:46:19.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:46:19.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:19.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:19.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:20.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:20.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:20.682 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:46:20.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:46:20.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:20.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:21.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:22.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:22.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:22.108 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:46:22.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:46:22.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:22.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:22.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:23.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:23.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:23.555 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:46:23.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:46:23.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:23.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:23.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:24.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:24.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:24.880 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:46:24.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:46:24.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:24.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:25.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:26.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:26.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:26.265 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:46:26.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:46:26.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:26.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:26.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:27.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:27.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:27.585 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:46:27.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:46:27.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:27.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:28.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:29.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:29.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:29.106 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:46:29.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:46:29.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:29.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:29.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:30.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:30.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:30.692 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:46:30.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:46:30.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:30.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:31.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:32.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:32.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:32.452 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:46:32.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:46:32.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:32.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:33.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:34.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:34.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:34.243 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:46:34.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:46:34.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:34.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:34.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:35.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:35.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:35.817 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:46:35.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:46:35.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:35.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:36.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:37.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:37.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:37.229 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:46:37.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:46:37.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:37.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:37.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:38.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:38.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:38.756 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:46:38.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:46:38.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:38.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:46:39.067 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2735536143,v1:127.0.0.1:6819/2735536143] [v2:127.0.0.1:6820/2735536143,v1:127.0.0.1:6821/2735536143] exists,up bd6a5d1f-3a4d-49e3-8791-f78ea09d9756 2024-03-29T05:46:39.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:46:39.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:46:39.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:46:39.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-03-29T05:46:39.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-29T05:46:39.544 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'testpool' created 2024-03-29T05:46:39.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:46:40.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-03-29T05:46:41.085 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 2 2024-03-29T05:46:41.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-29T05:46:41.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:46:41.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:46:41.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:46:41.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:46:41.108 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:46:41.108 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:46:41.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:46:41.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:46:41.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:46:41.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:46:41.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:46:41.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:46:41.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:46:41.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:46:41.231 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:46:41.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:46:41.730 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:46:41.730 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:46:41.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:46:41.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:46:41.730 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:46:41.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-29T05:46:41.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-29T05:46:41.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-29T05:46:41.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:46:41.878 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:46:42.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-29T05:46:42.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-29T05:46:42.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-03-29T05:46:42.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:46:42.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:46:42.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-29T05:46:42.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-29T05:46:42.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509449' 2024-03-29T05:46:42.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:46:42.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-29T05:46:42.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:46:42.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:46:42.166 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-29T05:46:42.166 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:46:42.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-29T05:46:42.167 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836498 2024-03-29T05:46:42.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-29T05:46:42.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:46:42.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-03-29T05:46:42.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:46:43.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:46:43.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:46:43.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-03-29T05:46:43.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:46:44.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T05:46:44.809 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:46:45.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836498 2024-03-29T05:46:45.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:46:45.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-29T05:46:45.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:46:45.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:46:45.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-29T05:46:45.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:46:45.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-29T05:46:45.115 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672976 2024-03-29T05:46:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-29T05:46:45.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:46:45.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-29T05:46:45.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:46:45.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-29T05:46:45.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:46:45.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:46:45.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-29T05:46:45.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:46:45.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-29T05:46:45.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-29T05:46:45.438 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509449 2024-03-29T05:46:45.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:46:45.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-29T05:46:45.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:46:45.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:46:45.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:46:46.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:46:46.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:46:46.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:46:46.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:46:46.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:46:46.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:46:46.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:46:46.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:46:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:46:46.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:46:46.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:46:46.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:46:46.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:46:46.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:46:46.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:46:46.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-03-29T05:46:46.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-03-29T05:46:46.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-03-29T05:46:46.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:46.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:46.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:46.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:47.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:47.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:47.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:47.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:47.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:47.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:47.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:47.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:47.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:47.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:47.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:47.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:47.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:47.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:48.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-29T05:46:48.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-29T05:46:48.173 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-29T05:46:48.174 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:46:48.174 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:46:48.174 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:46:48.174 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:46:48.174 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:46:48.175 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:46:48.175 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:46:48.498 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:46:48.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:46:48.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:46:48.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:46:48.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:46:48.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:46:48.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:46:48.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:46:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-29T05:46:50.332 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-29T05:46:50.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:46:50.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:46:50.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:46:50.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:46:50.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:46:50.872 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:46:50.872 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:46:50.872 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:46:50.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:46:50.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:46:50.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:46:50.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:46:50.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:46:50.875 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:46:50.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:46:50.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:46:50.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:46:50.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:46:50.910 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:50.912+0000 7ff4103f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:50.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:50.916+0000 7ff4103f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:50.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:50.916+0000 7ff4103f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:46:51.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:46:51.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:46:51.438 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:51.440+0000 7ff4103f27c0 -1 Falling back to public interface 2024-03-29T05:46:51.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:52.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:52.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:52.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:46:52.512 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:46:52.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:52.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:46:52.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:53.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:53.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:53.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:46:53.827 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:46:53.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:53.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:46:53.968 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:46:53.972+0000 7ff4103f27c0 -1 osd.0 21 log_to_monitors true 2024-03-29T05:46:54.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:46:55.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:46:55.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:46:55.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:46:55.170 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:46:55.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:46:55.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:46:55.483 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3074017929,v1:127.0.0.1:6803/3074017929] [v2:127.0.0.1:6804/3074017929,v1:127.0.0.1:6805/3074017929] exists,up e8149d32-5e6c-42c1-99df-4bf2d0d4fde3 2024-03-29T05:46:55.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:46:55.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:46:55.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:46:55.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:46:55.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:46:55.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:46:55.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:46:55.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:46:55.484 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:46:55.484 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:46:55.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:46:55.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:46:55.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:46:55.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:46:55.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:46:55.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:46:55.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:46:55.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:46:55.588 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:46:55.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:46:55.896 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:46:55.896 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:46:55.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:46:55.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:46:55.896 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:46:56.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-29T05:46:56.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-29T05:46:56.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-29T05:46:56.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:46:56.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:46:56.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-29T05:46:56.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-29T05:46:56.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-03-29T05:46:56.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:46:56.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:46:56.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-29T05:46:56.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-29T05:46:56.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509453' 2024-03-29T05:46:56.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:46:56.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-29T05:46:56.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:46:56.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:46:56.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-29T05:46:56.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:46:56.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-29T05:46:56.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-29T05:46:56.386 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 107374182402 2024-03-29T05:46:56.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:46:56.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-29T05:46:56.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:46:57.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:46:57.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:46:58.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-29T05:46:58.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:46:58.006 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-29T05:46:58.006 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:46:58.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:46:58.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:46:58.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-29T05:46:58.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-29T05:46:58.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-29T05:46:58.010 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672980 2024-03-29T05:46:58.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:46:58.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-29T05:46:58.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:46:58.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-29T05:46:58.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:46:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:46:58.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-29T05:46:58.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:46:58.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-29T05:46:58.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-29T05:46:58.339 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509453 2024-03-29T05:46:58.339 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:46:58.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-29T05:46:58.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:46:58.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:46:58.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:46:59.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:46:59.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:46:59.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:46:59.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:46:59.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:46:59.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:46:59.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:46:59.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:46:59.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:46:59.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:46:59.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:46:59.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:46:59.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:46:59.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:46:59.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:46:59.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-29T05:46:59.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:46:59.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:46:59.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:46:59.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:47:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:47:00.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:47:00.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:47:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:47:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:47:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:47:00.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-03-29T05:47:02.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:47:02.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:47:02.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:47:02.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:47:02.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:47:02.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:47:02.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:47:02.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:47:02.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:47:02.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:47:02.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:47:02.183 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:47:02.183 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:02.183 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:02.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:02.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:47:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:47:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:47:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:47:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:47:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:47:02.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:47:02.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:47:02.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:47:02.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:47:02.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:47:02.187 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:47:02.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:47:02.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:47:02.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:47:02.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:47:02.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:47:02.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:47:02.220 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:02.224+0000 7f2e325c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:02.222 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:02.224+0000 7f2e325c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:02.224 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:02.228+0000 7f2e325c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:02.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:47:02.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:47:02.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:47:02.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:47:02.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:47:02.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:02.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:47:02.507 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:47:02.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:02.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:02.758 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:02.760+0000 7f2e325c37c0 -1 Falling back to public interface 2024-03-29T05:47:02.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:03.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:03.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:03.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:47:03.840 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:03.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:03.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:04.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:05.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:47:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:47:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:05.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:05.492 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:05.496+0000 7f2e325c37c0 -1 osd.1 26 log_to_monitors true 2024-03-29T05:47:06.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:06.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:06.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:47:06.468 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:47:06.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:06.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:06.788 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/2180993854,v1:127.0.0.1:6811/2180993854] [v2:127.0.0.1:6812/2180993854,v1:127.0.0.1:6813/2180993854] exists,up 625fccf2-1a94-43cb-b66b-4b3c935de44d 2024-03-29T05:47:06.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:47:06.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:47:06.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:47:06.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:47:06.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:47:06.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:47:06.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:47:06.789 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:47:06.790 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:47:06.790 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:47:06.790 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:47:06.790 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:47:06.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:47:06.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:47:06.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:47:06.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:47:06.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:47:06.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:47:06.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:47:07.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:47:07.191 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:07.191 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:47:07.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:47:07.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:07.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:47:07.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-03-29T05:47:07.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-03-29T05:47:07.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-03-29T05:47:07.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:07.359 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:47:07.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-29T05:47:07.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-29T05:47:07.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-03-29T05:47:07.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:07.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:47:07.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-29T05:47:07.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-29T05:47:07.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509456' 2024-03-29T05:47:07.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:07.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-03-29T05:47:07.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:07.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:47:07.641 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-03-29T05:47:07.641 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:07.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-03-29T05:47:07.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-03-29T05:47:07.642 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 107374182405 2024-03-29T05:47:07.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:47:07.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-03-29T05:47:07.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:07.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-03-29T05:47:07.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:07.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:47:07.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-03-29T05:47:07.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:07.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-29T05:47:07.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-03-29T05:47:07.962 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 128849018882 2024-03-29T05:47:07.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:47:08.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-29T05:47:08.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:08.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-29T05:47:08.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:08.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:47:08.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-29T05:47:08.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:08.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-29T05:47:08.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-29T05:47:08.288 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509456 2024-03-29T05:47:08.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:47:08.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-29T05:47:08.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:47:08.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:08.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:08.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:47:08.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:47:08.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:47:08.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:47:08.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:47:08.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:47:08.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:47:08.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:47:09.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:47:09.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:47:09.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:09.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:09.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:47:09.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:47:09.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:47:09.606 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-29T05:47:09.606 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:47:09.606 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:47:09.606 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:47:09.606 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:47:09.606 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:47:09.607 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:47:09.607 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:47:09.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:47:09.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:47:09.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:47:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:47:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:47:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:47:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:47:10.233 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:47:10.233 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:47:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:47:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:47:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:47:10.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-29T05:47:11.709 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:104778fc:::obj2:head# 2024-03-29T05:47:12.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:47:12.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:47:12.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:47:12.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:47:12.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:47:12.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:12.253 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:12.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:12.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:47:12.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:47:12.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:47:12.255 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:47:12.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:47:12.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:47:12.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:47:12.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:47:12.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:47:12.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:47:12.292 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:12.296+0000 7f7abe3ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:12.310 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:12.312+0000 7f7abe3ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:12.311 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:12.312+0000 7f7abe3ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:12.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:47:12.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:47:12.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:47:12.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:47:12.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:47:12.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:12.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:47:12.575 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:47:12.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:12.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:47:12.846 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:12.848+0000 7f7abe3ec7c0 -1 Falling back to public interface 2024-03-29T05:47:12.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:13.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:13.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:13.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:47:13.895 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:13.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:13.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:47:14.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:47:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:47:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:47:15.273 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:15.276+0000 7f7abe3ec7c0 -1 osd.0 31 log_to_monitors true 2024-03-29T05:47:15.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:16.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:16.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:16.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:47:16.521 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:47:16.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:16.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:47:16.831 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/1448395483,v1:127.0.0.1:6803/1448395483] [v2:127.0.0.1:6804/1448395483,v1:127.0.0.1:6805/1448395483] exists,up e8149d32-5e6c-42c1-99df-4bf2d0d4fde3 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:47:16.832 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:47:16.833 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:47:16.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:47:16.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:47:16.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:47:16.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:47:16.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:47:16.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:47:16.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:47:16.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:47:16.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:47:17.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:47:17.217 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:17.217 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:47:17.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:47:17.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:17.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:47:17.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-29T05:47:17.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-29T05:47:17.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-29T05:47:17.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:17.379 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:47:17.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-29T05:47:17.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-29T05:47:17.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-03-29T05:47:17.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:17.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:47:17.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-29T05:47:17.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-29T05:47:17.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509459' 2024-03-29T05:47:17.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:17.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-29T05:47:17.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:17.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:47:17.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-29T05:47:17.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:17.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-29T05:47:17.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-29T05:47:17.662 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 150323855362 2024-03-29T05:47:17.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:47:17.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-29T05:47:17.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:17.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018885 2024-03-29T05:47:17.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:17.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:47:17.991 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018885 2024-03-29T05:47:17.991 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:17.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-29T05:47:17.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-03-29T05:47:17.992 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 128849018885 2024-03-29T05:47:17.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:47:18.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-29T05:47:18.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:18.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-29T05:47:18.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:18.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:47:18.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-29T05:47:18.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:18.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-29T05:47:18.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-29T05:47:18.352 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509459 2024-03-29T05:47:18.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:47:18.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-29T05:47:18.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:47:18.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:18.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:19.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:47:19.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:47:19.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:47:19.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:47:19.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:47:19.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:47:19.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:47:19.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:47:19.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:47:19.421 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:47:19.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:19.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:47:19.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:47:20.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:47:20.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:47:20.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:47:20.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:47:20.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:47:20.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:47:20.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-29T05:47:20.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:47:20.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:47:20.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:47:20.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:47:20.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:47:20.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-29T05:47:21.498 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:47:22.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:22.039 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:22.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:22.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:47:22.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:47:22.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:47:22.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:47:22.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:47:22.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:47:22.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:47:22.076 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:22.080+0000 7f97e5e657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:22.079 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:22.080+0000 7f97e5e657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:22.080 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:22.084+0000 7f97e5e657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:22.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:47:22.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:47:22.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:47:22.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:47:22.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:47:22.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:22.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:47:22.394 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:47:22.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:22.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:22.613 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:22.616+0000 7f97e5e657c0 -1 Falling back to public interface 2024-03-29T05:47:22.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:23.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:23.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:23.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:47:23.710 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:23.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:23.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:24.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:25.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:25.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:25.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:47:25.011 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:47:25.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:25.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:25.061 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:25.064+0000 7f97e5e657c0 -1 osd.1 36 log_to_monitors true 2024-03-29T05:47:25.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:47:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:47:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:47:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:47:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:47:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:47:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:47:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/4080930110,v1:127.0.0.1:6811/4080930110] [v2:127.0.0.1:6812/4080930110,v1:127.0.0.1:6813/4080930110] exists,up 625fccf2-1a94-43cb-b66b-4b3c935de44d 2024-03-29T05:47:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:47:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:47:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:47:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:47:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:47:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:47:26.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:47:26.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:47:26.671 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:47:26.671 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:47:26.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:47:26.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:47:26.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:47:26.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:47:26.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:47:26.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:47:26.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:47:26.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:47:26.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:47:27.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:47:27.091 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:27.091 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:47:27.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:47:27.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:27.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:47:27.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-29T05:47:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-29T05:47:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365' 2024-03-29T05:47:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:27.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:47:27.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-29T05:47:27.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-29T05:47:27.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-03-29T05:47:27.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:27.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:47:27.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-29T05:47:27.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-29T05:47:27.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509462' 2024-03-29T05:47:27.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:27.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:27.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855365 2024-03-29T05:47:27.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:47:27.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855365 2024-03-29T05:47:27.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:27.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-29T05:47:27.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-03-29T05:47:27.541 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 150323855365 2024-03-29T05:47:27.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:47:27.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855365 2024-03-29T05:47:27.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:47:28.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:47:28.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:47:29.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-29T05:47:29.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:29.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-29T05:47:29.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:29.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:47:29.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-29T05:47:29.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:29.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-29T05:47:29.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-29T05:47:29.154 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 171798691842 2024-03-29T05:47:29.154 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:47:29.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-29T05:47:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:29.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-29T05:47:29.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:29.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:47:29.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-29T05:47:29.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:29.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-29T05:47:29.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-29T05:47:29.476 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509462 2024-03-29T05:47:29.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:47:29.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-29T05:47:29.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:47:29.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:29.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:30.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:47:30.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:47:30.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:47:30.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:47:30.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:47:30.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:47:30.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:47:30.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:47:30.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:47:30.535 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:47:30.535 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:30.535 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:30.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:47:30.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:47:30.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:47:30.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-29T05:47:31.050 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0: { 2024-03-29T05:47:31.050 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-29T05:47:31.050 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:47:31.063 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1: { 2024-03-29T05:47:31.063 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-29T05:47:31.064 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:47:31.076 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2: { 2024-03-29T05:47:31.076 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-29T05:47:31.076 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:47:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-03-29T05:47:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-29T05:47:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-29T05:47:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:47:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-29T05:47:31.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-03-29T05:47:31.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-03-29T05:47:31.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:47:31.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-29T05:47:31.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-29T05:47:31.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:47:31.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-03-29T05:47:31.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-03-29T05:47:31.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:31.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:31.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:31.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:32.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:32.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-03-29T05:47:32.182 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:47:32.182 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": true, 2024-03-29T05:47:32.183 INFO:tasks.workunit.client.0.smithi123.stdout: "must": false, 2024-03-29T05:47:32.183 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "2024-03-15T05:45:52.189743+0000" 2024-03-29T05:47:32.183 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:47:32.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:32.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:32.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:32.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:46:39.442390+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:32.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:33.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:33.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:33.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:33.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:33.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:33.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:33.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:33.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:46:39.442390+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:33.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:34.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:34.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:34.784 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:34.784 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:34.784 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:34.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:34.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:35.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:46:39.442390+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:35.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:36.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:36.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:36.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:36.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:36.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:36.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:36.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:36.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-15T05:45:52.189743+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:36.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:37.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:37.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:37.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:37.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:37.416 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:37.416 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:37.416 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:37.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-15T05:45:52.189743+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:37.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:38.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:38.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:38.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:38.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:38.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:38.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:39.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-15T05:45:52.189743+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:39.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:40.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:40.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:40.058 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:40.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:40.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:40.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:40.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:40.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-15T05:45:52.189743+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:40.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:41.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:41.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:41.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:41.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:41.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:41.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:41.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:41.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-15T05:45:52.189743+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:41.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:47:42.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:47:42.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:47:42.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:47:42.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:47:42.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:47:42.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:47:42.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:47:42.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:47:36.195004+0000 '>' 2024-03-29T05:46:39.442390+0000 2024-03-29T05:47:42.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:47:42.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-29T05:47:42.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:47:42.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:47:42.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:47:42.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:47:42.978 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:47:42.978 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:47:42.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:47:42.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:47:42.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:47:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:47:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:47:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:47:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:47:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:47:43.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:47:43.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:47:43.427 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:43.427 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:47:43.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:47:43.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:43.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:47:43.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855369 2024-03-29T05:47:43.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855369 2024-03-29T05:47:43.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369' 2024-03-29T05:47:43.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:43.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:47:43.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691846 2024-03-29T05:47:43.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691846 2024-03-29T05:47:43.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369 1-171798691846' 2024-03-29T05:47:43.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:43.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:47:43.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-29T05:47:43.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-29T05:47:43.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855369 1-171798691846 2-64424509467' 2024-03-29T05:47:43.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:43.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855369 2024-03-29T05:47:43.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:43.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:47:43.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855369 2024-03-29T05:47:43.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:43.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855369 2024-03-29T05:47:43.846 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 150323855369 2024-03-29T05:47:43.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855369' 2024-03-29T05:47:43.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:47:44.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855369 -lt 150323855369 2024-03-29T05:47:44.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:44.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691846 2024-03-29T05:47:44.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:47:44.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691846 2024-03-29T05:47:44.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:44.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691846 2024-03-29T05:47:44.156 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 171798691846 2024-03-29T05:47:44.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-03-29T05:47:44.157 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:47:44.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691846 2024-03-29T05:47:44.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:44.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-29T05:47:44.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:44.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:47:44.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-29T05:47:44.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:44.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-29T05:47:44.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-29T05:47:44.485 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509467 2024-03-29T05:47:44.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:47:44.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509467 2024-03-29T05:47:44.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:47:45.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:47:45.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:47:46.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-29T05:47:46.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:47:46.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:46.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:46.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:47:46.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:47:46.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:47:46.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:47:46.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:47:46.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:47:46.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:47:46.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:47:46.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:47:46.809 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:47:46.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:47:46.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:47:47.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:47:47.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:47:47.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:47:47.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-03-29T05:47:47.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:47:47.154 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:47:47.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:47:47.493 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:47:47.493 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:47:47.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:47:47.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:47.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:47:47.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855371 2024-03-29T05:47:47.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855371 2024-03-29T05:47:47.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371' 2024-03-29T05:47:47.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:47.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:47:47.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691848 2024-03-29T05:47:47.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691848 2024-03-29T05:47:47.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371 1-171798691848' 2024-03-29T05:47:47.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:47:47.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:47:47.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-03-29T05:47:47.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-03-29T05:47:47.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855371 1-171798691848 2-64424509468' 2024-03-29T05:47:47.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:47.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855371 2024-03-29T05:47:47.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:47.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:47:47.905 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855371 2024-03-29T05:47:47.905 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:47.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855371 2024-03-29T05:47:47.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855371' 2024-03-29T05:47:47.907 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 150323855371 2024-03-29T05:47:47.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:47:48.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855371 -lt 150323855371 2024-03-29T05:47:48.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:48.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691848 2024-03-29T05:47:48.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:48.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:47:48.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691848 2024-03-29T05:47:48.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:48.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691848 2024-03-29T05:47:48.203 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 171798691848 2024-03-29T05:47:48.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691848' 2024-03-29T05:47:48.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:47:48.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691848 2024-03-29T05:47:48.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:47:49.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:47:49.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:47:49.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691848 2024-03-29T05:47:49.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:47:50.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T05:47:50.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:47:51.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691848 -lt 171798691848 2024-03-29T05:47:51.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:47:51.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-03-29T05:47:51.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:47:51.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:47:51.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-03-29T05:47:51.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:47:51.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-03-29T05:47:51.150 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509468 2024-03-29T05:47:51.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-03-29T05:47:51.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:47:51.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509468 2024-03-29T05:47:51.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-03-29T05:47:51.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-29T05:47:51.782 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:47:51.782 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-29T05:47:36.195218+0000 21'10 41:97 [1,0] 1 [1,0] 1 21'10 2024-03-29T05:47:36.195004+0000 21'10 2024-03-29T05:47:36.195004+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:47:36.195004+0000 10 0 2024-03-29T05:47:51.782 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:47:51.782 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:47:51.783 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:47:51.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-29T05:47:51.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-03-29T05:47:52.070 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:47:52.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:47:52.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:47:52.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:47:52.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:47:52.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:47:52.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:47:52.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:47:52.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:47:52.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:47:52.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:47:52.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:47:52.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:47:52.400 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:47:52.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:47:52.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:47:52.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:47:52.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:47:52.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:47:52.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:47:52.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:47:52.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:47:52.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:47:52.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:47:52.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:47:52.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:52.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:52.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:47:52.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:47:52.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:47:52.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:47:52.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:47:52.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:47:52.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:47:52.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:47:52.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:47:52.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:47:52.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:47:52.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:47:52.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:47:52.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:47:52.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:47:52.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:47:52.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:47:52.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:47:52.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:47:52.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:47:52.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:47:52.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:47:52.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:52.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:52.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:47:52.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:47:52.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:47:52.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:47:52.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:47:52.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:52.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:52.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:47:52.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:47:52.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:47:52.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:47:52.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T05:47:52.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:47:52.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:47:52.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:47:52.843 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:47:52.843 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:52.843 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:52.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:52.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T05:47:53.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:47:53.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:47:53.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:47:53.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:47:53.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:47:53.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:47:53.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:47:53.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:47:53.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:47:53.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:47:53.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:47:53.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:47:53.181 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:47:53.181 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:47:53.181 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:47:53.181 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:47:53.181 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:53.181 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:53.181 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:47:53.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:47:53.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:47:53.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-03-29T05:47:53.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:47:53.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:47:53.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:47:53.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:47:53.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:47:53.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:47:54.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:47:54.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:47:54.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:47:54.236 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:47:54.237 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:54.237 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:54.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:54.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:47:54.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:47:54.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-29T05:47:54.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-03-29T05:47:54.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:47:54.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:47:54.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:47:54.280 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:47:54.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2025209-99bd-45ef-9487-8706ce2fe725 2024-03-29T05:47:54.281 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 e2025209-99bd-45ef-9487-8706ce2fe725 2024-03-29T05:47:54.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e2025209-99bd-45ef-9487-8706ce2fe725' 2024-03-29T05:47:54.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:47:54.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKVgZmmfIdEhAAyxyneAmN8h6s3gd49cG/Mg== 2024-03-29T05:47:54.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKVgZmmfIdEhAAyxyneAmN8h6s3gd49cG/Mg=="}' 2024-03-29T05:47:54.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2025209-99bd-45ef-9487-8706ce2fe725 -i td/osd-scrub-repair/0/new.json 2024-03-29T05:47:54.530 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:47:54.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:47:54.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQAKVgZmmfIdEhAAyxyneAmN8h6s3gd49cG/Mg== --osd-uuid e2025209-99bd-45ef-9487-8706ce2fe725 2024-03-29T05:47:54.587 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:54.587+0000 7f52f3ccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:54.591+0000 7f52f3ccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:54.591+0000 7f52f3ccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:47:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:54.591+0000 7f52f3ccb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:47:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:54.595+0000 7f52f3ccb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:47:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:54.595+0000 7f52f3ccb7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:47:54.673 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:47:54.675+0000 7f52f3ccb7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:48:01.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:48:01.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:48:01.044 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:48:01.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:48:01.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:48:01.435 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:48:01.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:48:01.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:48:01.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:48:01.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:48:01.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:48:01.471 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:01.475+0000 7feeb6d907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:01.504 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:01.507+0000 7feeb6d907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:01.506 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:01.507+0000 7feeb6d907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:01.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:48:01.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:48:01.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:48:01.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:48:01.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:48:01.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:01.895 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:48:01.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:48:01.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:01.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:02.073 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:02.075+0000 7feeb6d907c0 -1 Falling back to public interface 2024-03-29T05:48:02.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:03.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:03.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:03.233 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:48:03.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:48:03.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:03.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:03.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:04.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:04.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:04.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:48:04.550 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:48:04.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:04.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:04.706 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:04.707+0000 7feeb6d907c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:48:04.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:05.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:05.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:05.903 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:48:05.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:48:05.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:05.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:06.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:07.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:07.226 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:48:07.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:07.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:48:07.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:07.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:07.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:08.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:08.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:08.550 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:48:08.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:48:08.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:08.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:08.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:09.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:09.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:09.871 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:48:09.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:48:09.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:09.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:10.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:11.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:11.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:11.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:48:11.181 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:48:11.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:11.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:11.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:12.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:12.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:12.556 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:48:12.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:48:12.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:12.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:12.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:13.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:13.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:13.873 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:48:13.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:48:13.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:13.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:14.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:15.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:15.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:15.184 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:48:15.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:48:15.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:15.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:15.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:16.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:16.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:16.585 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:48:16.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:48:16.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:16.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:16.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:17.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:17.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:17.897 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:48:17.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:48:17.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:17.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:18.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:19.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:19.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:19.201 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:48:19.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:48:19.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:19.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:19.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:20.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:20.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:20.916 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:48:20.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:48:20.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:20.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:21.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:22.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:22.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:22.438 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:48:22.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:48:22.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:22.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:23.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:24.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:24.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:24.033 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:48:24.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:48:24.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:24.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:24.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:25.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:25.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:25.356 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:48:25.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:48:25.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:25.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:26.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:27.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:27.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:27.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:48:27.023 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:48:27.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:27.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:27.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:28.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:28.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:28.612 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:48:28.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:48:28.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:28.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:29.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:30.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:30.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:30.113 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:48:30.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:48:30.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:30.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:48:30.429 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1438904478,v1:127.0.0.1:6803/1438904478] [v2:127.0.0.1:6804/1438904478,v1:127.0.0.1:6805/1438904478] exists,up e2025209-99bd-45ef-9487-8706ce2fe725 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:48:30.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:48:30.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:48:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:48:30.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:48:30.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-29T05:48:30.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:48:30.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:48:30.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=417fa382-7f62-458d-9d88-a92b52f819ad 2024-03-29T05:48:30.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 417fa382-7f62-458d-9d88-a92b52f819ad' 2024-03-29T05:48:30.437 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 417fa382-7f62-458d-9d88-a92b52f819ad 2024-03-29T05:48:30.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:48:30.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuVgZmQzmzGxAAqNSUiPXu5qroJum5uyxPEQ== 2024-03-29T05:48:30.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuVgZmQzmzGxAAqNSUiPXu5qroJum5uyxPEQ=="}' 2024-03-29T05:48:30.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 417fa382-7f62-458d-9d88-a92b52f819ad -i td/osd-scrub-repair/1/new.json 2024-03-29T05:48:30.842 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:48:30.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:48:30.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQAuVgZmQzmzGxAAqNSUiPXu5qroJum5uyxPEQ== --osd-uuid 417fa382-7f62-458d-9d88-a92b52f819ad 2024-03-29T05:48:30.901 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:30.906+0000 7f61e9fc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:30.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:30.958+0000 7f61e9fc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:30.954 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:30.958+0000 7f61e9fc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:30.955 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:30.958+0000 7f61e9fc97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:48:30.955 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:30.958+0000 7f61e9fc97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:48:30.955 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:30.958+0000 7f61e9fc97c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:48:31.026 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:31.030+0000 7f61e9fc97c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:48:38.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:48:38.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:48:38.146 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:48:38.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:48:38.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:48:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:48:38.523 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:48:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:48:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:48:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:48:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:48:38.558 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:38.561+0000 7f23948b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:38.590 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:38.593+0000 7f23948b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:38.591 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:38.593+0000 7f23948b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:48:38.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:48:38.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:48:38.891 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:48:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:48:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:48:38.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:48:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:48:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:38.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:39.154 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:39.157+0000 7f23948b77c0 -1 Falling back to public interface 2024-03-29T05:48:39.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:40.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:40.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:40.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:48:40.226 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:48:40.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:40.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:40.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:41.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:41.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:41.524 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:48:41.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:48:41.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:41.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:41.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:41.952 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:48:41.957+0000 7f23948b77c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:48:42.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:42.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:42.836 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:48:42.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:48:42.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:42.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:43.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:44.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:44.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:44.305 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:48:44.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:48:44.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:44.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:44.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:45.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:45.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:45.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:48:45.620 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:48:45.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:45.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:46.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:47.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:47.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:47.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:48:47.005 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:48:47.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:47.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:47.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:48.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:48.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:48.316 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:48:48.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:48:48.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:48.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:48.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:49.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:49.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:49.628 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:48:49.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:48:49.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:49.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:50.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:51.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:51.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:51.092 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:48:51.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:48:51.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:51.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:51.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:52.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:52.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:52.431 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:48:52.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:48:52.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:52.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:52.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:53.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:53.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:53.745 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:48:53.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:48:53.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:53.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:54.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:55.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:55.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:55.179 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:48:55.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:48:55.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:55.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:55.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:56.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:56.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:56.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:48:56.655 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:48:56.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:56.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:57.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:48:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:48:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:48:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:48:58.797 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:48:58.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:48:58.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:48:59.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:00.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:00.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:00.607 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:49:00.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:49:00.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:00.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:49:01.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:02.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:02.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:02.040 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:49:02.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:49:02.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:02.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:49:02.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:03.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:03.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:03.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:49:03.659 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:49:03.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:03.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:49:04.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:05.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:05.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:05.453 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:49:05.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:49:05.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:05.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:49:05.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:06.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:06.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:06.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:49:06.931 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:49:06.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:06.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:49:07.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:08.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:08.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:08.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:49:08.325 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:49:08.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:08.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:49:08.642 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1654241031,v1:127.0.0.1:6811/1654241031] [v2:127.0.0.1:6812/1654241031,v1:127.0.0.1:6813/1654241031] exists,up 417fa382-7f62-458d-9d88-a92b52f819ad 2024-03-29T05:49:08.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:49:08.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:49:08.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:49:08.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-29T05:49:08.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:49:08.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:49:08.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T05:49:08.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:49:08.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-29T05:49:08.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T05:49:08.649 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:49:08.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e41287a-a55d-450d-bfff-c896f2c18ace 2024-03-29T05:49:08.649 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 9e41287a-a55d-450d-bfff-c896f2c18ace 2024-03-29T05:49:08.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9e41287a-a55d-450d-bfff-c896f2c18ace' 2024-03-29T05:49:08.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:49:08.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUVgZmksA8KBAADGM2YJZxH8lZ1UZ6O8k+Gw== 2024-03-29T05:49:08.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUVgZmksA8KBAADGM2YJZxH8lZ1UZ6O8k+Gw=="}' 2024-03-29T05:49:08.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e41287a-a55d-450d-bfff-c896f2c18ace -i td/osd-scrub-repair/2/new.json 2024-03-29T05:49:09.040 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:49:09.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T05:49:09.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBUVgZmksA8KBAADGM2YJZxH8lZ1UZ6O8k+Gw== --osd-uuid 9e41287a-a55d-450d-bfff-c896f2c18ace 2024-03-29T05:49:09.101 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:09.104+0000 7fac82bd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:09.141 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:09.144+0000 7fac82bd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:09.142 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:09.144+0000 7fac82bd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:09.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:09.144+0000 7fac82bd87c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:49:09.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:09.144+0000 7fac82bd87c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:49:09.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:09.144+0000 7fac82bd87c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:49:09.211 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:09.212+0000 7fac82bd87c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:49:16.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T05:49:16.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:49:16.291 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:49:16.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:49:16.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:49:16.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:49:16.677 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:49:16.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:49:16.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:49:16.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:49:16.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:49:16.712 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:16.716+0000 7f41d018c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:16.738 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:16.740+0000 7f41d018c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:16.739 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:16.740+0000 7f41d018c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:49:17.009 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:49:17.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:17.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:17.294 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:17.296+0000 7f41d018c7c0 -1 Falling back to public interface 2024-03-29T05:49:17.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:18.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:18.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:18.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:49:18.319 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:49:18.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:18.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:18.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:19.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:19.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:19.638 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:49:19.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:49:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:19.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:19.975 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:19.976+0000 7f41d018c7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:49:20.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:20.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:20.947 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:49:20.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:49:20.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:20.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:21.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:22.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:22.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:22.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:49:22.471 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:49:22.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:22.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:22.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:23.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:23.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:23.937 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:49:23.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:49:23.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:23.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:24.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:25.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:25.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:25.261 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:49:25.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:49:25.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:25.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:25.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:26.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:26.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:26.670 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:49:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:49:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:26.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:27.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:28.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:28.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:28.091 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:49:28.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:49:28.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:28.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:28.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:29.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:29.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:29.411 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:49:29.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:49:29.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:29.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:29.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:30.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:30.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:30.858 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:49:30.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:49:30.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:30.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:31.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:32.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:32.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:32.176 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:49:32.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:49:32.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:32.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:32.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:33.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:33.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:33.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:49:33.497 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:49:33.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:33.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:33.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:34.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:34.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:34.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:49:34.924 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:49:34.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:34.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:35.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:36.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:36.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:36.895 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:49:36.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:49:36.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:36.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:37.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:38.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:38.429 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:49:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:49:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:38.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:38.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:39.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:39.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:39.890 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:49:39.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:49:39.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:39.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:40.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:41.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:41.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:41.349 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:49:41.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:49:41.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:41.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:42.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:43.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:43.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:43.133 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:49:43.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:49:43.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:43.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:43.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:44.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:44.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:44.773 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:49:44.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:49:44.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:44.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:45.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:49:46.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:49:46.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:46.321 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:49:46.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:49:46.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:46.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:49:46.651 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/913236334,v1:127.0.0.1:6819/913236334] [v2:127.0.0.1:6820/913236334,v1:127.0.0.1:6821/913236334] exists,up 9e41287a-a55d-450d-bfff-c896f2c18ace 2024-03-29T05:49:46.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:49:46.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:49:46.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:49:46.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-03-29T05:49:46.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-29T05:49:47.272 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'testpool' created 2024-03-29T05:49:47.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:49:48.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-03-29T05:49:48.693 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 2 2024-03-29T05:49:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-29T05:49:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:49:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:49:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:49:48.710 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:49:48.710 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:49:48.710 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:49:48.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:49:48.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:49:48.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:49:48.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:49:48.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:49:48.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:49:48.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:49:48.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:49:48.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:49:49.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:49:49.163 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:49:49.163 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:49:49.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:49:49.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:49:49.163 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:49:49.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-29T05:49:49.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-29T05:49:49.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-29T05:49:49.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:49:49.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:49:49.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-29T05:49:49.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-29T05:49:49.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672978' 2024-03-29T05:49:49.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:49:49.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:49:49.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-29T05:49:49.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-29T05:49:49.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672978 2-64424509447' 2024-03-29T05:49:49.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:49:49.773 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:49:49.773 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-29T05:49:49.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:49:49.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-29T05:49:49.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:49:49.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-29T05:49:49.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-29T05:49:49.775 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836506 2024-03-29T05:49:49.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:49:50.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836506 2024-03-29T05:49:50.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:49:51.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:49:51.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:49:51.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836506 2024-03-29T05:49:51.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:49:51.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-29T05:49:51.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:49:51.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:49:51.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-29T05:49:51.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:49:51.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-29T05:49:51.395 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672978 2024-03-29T05:49:51.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-29T05:49:51.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:49:51.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-29T05:49:51.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:49:51.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-29T05:49:51.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:49:51.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:49:51.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-29T05:49:51.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:49:51.685 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509447 2024-03-29T05:49:51.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-29T05:49:51.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-29T05:49:51.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:49:52.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-29T05:49:52.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:49:52.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:49:52.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:49:52.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:49:52.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:49:52.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:49:52.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:49:52.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:49:52.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:49:52.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:49:52.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:49:52.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T05:49:52.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:49:52.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:49:52.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:49:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-29T05:49:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-29T05:49:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T05:49:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-29T05:49:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:49:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T05:49:53.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:49:53.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:49:53.235 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:49:53.235 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:49:53.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:49:53.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:49:53.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:49:53.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:49:53.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T05:49:53.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:49:53.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:49:53.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:49:53.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-29T05:49:53.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-29T05:49:53.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T05:49:53.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T05:49:53.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T05:49:53.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T05:49:53.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T05:49:53.922 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T05:49:53.922 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T05:49:54.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T05:49:54.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T05:49:54.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T05:49:54.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:49:54.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T05:49:54.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:49:54.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:49:54.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:49:54.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:49:54.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:49:54.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:49:54.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:49:54.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:49:54.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:49:54.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:49:54.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:49:54.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:49:55.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:49:55.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:49:55.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:49:55.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-03-29T05:49:55.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-03-29T05:49:55.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T05:49:55.327 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-29T05:49:55.327 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:49:55.327 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:49:55.328 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:49:55.328 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:49:55.328 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:49:55.328 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:49:55.329 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:49:55.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:49:55.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:49:55.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:49:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:49:56.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:49:56.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:49:56.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:49:56.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:49:56.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:49:56.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:49:56.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:49:56.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:49:56.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:49:56.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:49:56.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:49:56.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:49:56.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:49:56.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:49:56.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-29T05:49:58.019 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:49:58.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:49:58.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:49:58.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:49:58.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:49:58.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:49:58.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:49:58.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:49:58.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:49:58.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:49:58.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:49:58.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:49:58.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-29T05:49:58.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:49:58.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:49:58.567 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:49:58.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:49:58.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:49:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:49:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:49:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:49:58.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:49:58.611 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:58.611+0000 7f3bb2c357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:58.614 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:58.615+0000 7f3bb2c357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:58.615 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:58.615+0000 7f3bb2c357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:49:58.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:49:58.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:49:58.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:49:58.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:49:58.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:49:58.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:49:58.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:49:58.903 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:49:58.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:49:58.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:49:59.151 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:49:59.151+0000 7f3bb2c357c0 -1 Falling back to public interface 2024-03-29T05:49:59.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:00.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:00.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:00.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:50:00.218 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:50:00.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:00.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:00.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:01.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:01.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:01.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:50:01.539 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:50:01.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:01.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:01.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:01.890 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:01.891+0000 7f3bb2c357c0 -1 osd.0 21 log_to_monitors true 2024-03-29T05:50:02.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:02.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:02.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:50:02.859 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:50:02.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:02.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4013148951,v1:127.0.0.1:6803/4013148951] [v2:127.0.0.1:6804/4013148951,v1:127.0.0.1:6805/4013148951] exists,up e2025209-99bd-45ef-9487-8706ce2fe725 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:50:03.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:50:03.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:50:03.217 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:50:03.217 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:50:03.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:50:03.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:50:03.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:50:03.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:50:03.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:50:03.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:50:03.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:50:03.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:50:03.326 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:50:03.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:50:03.667 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:50:03.667 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:50:03.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:50:03.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:03.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:50:03.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-29T05:50:03.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-29T05:50:03.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-29T05:50:03.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:03.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:50:03.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-29T05:50:03.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-29T05:50:03.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672982' 2024-03-29T05:50:03.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:03.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:50:04.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-29T05:50:04.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-29T05:50:04.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672982 2-64424509451' 2024-03-29T05:50:04.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:04.117 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-29T05:50:04.117 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:04.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:50:04.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-29T05:50:04.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:04.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-29T05:50:04.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-29T05:50:04.120 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 107374182402 2024-03-29T05:50:04.120 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:50:04.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-29T05:50:04.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:50:05.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:50:05.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:50:05.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-29T05:50:05.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:05.725 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:05.725 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-29T05:50:05.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:50:05.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-29T05:50:05.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:05.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-29T05:50:05.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-29T05:50:05.728 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672982 2024-03-29T05:50:05.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:50:06.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-03-29T05:50:06.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:06.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:06.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-29T05:50:06.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:50:06.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-29T05:50:06.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:06.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-29T05:50:06.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-29T05:50:06.031 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509451 2024-03-29T05:50:06.032 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:50:06.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-03-29T05:50:06.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:50:06.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:50:06.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:50:06.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:50:06.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:50:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:50:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:50:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:50:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:50:06.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:50:06.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:50:07.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:50:07.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:50:07.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:50:07.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:50:07.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:50:07.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:50:07.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:50:07.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-03-29T05:50:07.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-29T05:50:07.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T05:50:07.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-29T05:50:07.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:50:07.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-03-29T05:50:07.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-03-29T05:50:07.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:50:07.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:50:07.805 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:50:07.805 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:50:08.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-03-29T05:50:08.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-03-29T05:50:08.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:50:08.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:50:08.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:50:08.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:50:08.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:08.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-03-29T05:50:08.579 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T05:50:08.579 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": false, 2024-03-29T05:50:08.579 INFO:tasks.workunit.client.0.smithi123.stdout: "must": false, 2024-03-29T05:50:08.579 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "2024-03-22T05:48:28.585023+0000" 2024-03-29T05:50:08.579 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:50:08.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:50:08.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:50:08.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-22T05:48:28.585023+0000 '>' 2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:08.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:50:09.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:50:09.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:50:09.890 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:50:09.891 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:50:09.891 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:50:09.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:50:09.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:50:10.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-22T05:48:28.585023+0000 '>' 2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:10.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:50:11.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:50:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:50:11.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:50:11.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:50:11.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:50:11.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:50:11.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:50:11.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-22T05:48:28.585023+0000 '>' 2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:11.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:50:12.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:50:12.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:50:12.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:50:12.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:50:12.514 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:50:12.514 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:50:12.514 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:50:12.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-22T05:48:28.585023+0000 '>' 2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:50:13.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:50:13.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:50:13.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:50:13.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:50:13.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:50:13.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:50:13.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:50:14.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-22T05:48:28.585023+0000 '>' 2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:14.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:50:15.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:50:15.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:50:15.152 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T05:50:15.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:50:15.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:50:15.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:50:15.154 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:50:15.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:50:10.565664+0000 '>' 2024-03-29T05:49:47.118877+0000 2024-03-29T05:50:15.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:50:15.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-29T05:50:15.734 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:50:15.734 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-29T05:50:10.752853+0000 21'1 26:60 [1,0] 1 [1,0] 1 21'1 2024-03-29T05:50:10.565664+0000 21'1 2024-03-29T05:50:10.565664+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:50:10.565664+0000 1 0 2024-03-29T05:50:15.734 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:50:15.735 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:50:15.735 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:50:15.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-29T05:50:17.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-29T05:50:18.036 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:50:18.036 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-29T05:50:10.752853+0000 21'1 26:60 [1,0] 1 [1,0] 1 21'1 2024-03-29T05:50:10.565664+0000 21'1 2024-03-29T05:50:10.565664+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:50:10.565664+0000 1 0 2024-03-29T05:50:18.036 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:50:18.036 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:50:18.037 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:50:18.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-29T05:50:20.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-29T05:50:20.344 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:50:20.344 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-29T05:50:10.752853+0000 21'1 26:60 [1,0] 1 [1,0] 1 21'1 2024-03-29T05:50:10.565664+0000 21'1 2024-03-29T05:50:10.565664+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:50:10.565664+0000 1 0 2024-03-29T05:50:20.344 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:50:20.344 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:50:20.344 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:50:20.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-03-29T05:50:25.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-29T05:50:25.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:50:25.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:50:25.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:50:25.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:50:25.363 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:50:25.363 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:50:25.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:50:25.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:50:25.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:50:25.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:50:25.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:50:25.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:50:25.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:50:25.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:50:25.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:50:25.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:50:25.805 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:50:25.806 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:50:25.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:50:25.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:25.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:50:25.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-03-29T05:50:25.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-03-29T05:50:25.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-03-29T05:50:25.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:25.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:50:26.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-03-29T05:50:26.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-03-29T05:50:26.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672987' 2024-03-29T05:50:26.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:26.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:50:26.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-29T05:50:26.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-29T05:50:26.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672987 2-64424509456' 2024-03-29T05:50:26.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:26.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-03-29T05:50:26.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:26.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:50:26.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-03-29T05:50:26.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:26.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-03-29T05:50:26.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-03-29T05:50:26.241 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 107374182407 2024-03-29T05:50:26.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:50:26.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182407 2024-03-29T05:50:26.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:50:27.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:50:27.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:50:27.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-03-29T05:50:27.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:27.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-03-29T05:50:27.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:27.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:50:27.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-03-29T05:50:27.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:27.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-03-29T05:50:27.859 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672987 2024-03-29T05:50:27.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-03-29T05:50:27.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:50:28.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-03-29T05:50:28.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:28.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:28.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-29T05:50:28.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:50:28.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:28.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-29T05:50:28.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-29T05:50:28.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-29T05:50:28.212 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509456 2024-03-29T05:50:28.212 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:50:28.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-03-29T05:50:28.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:50:28.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:50:28.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:50:28.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:50:28.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:50:28.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:50:28.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:50:28.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:50:28.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:50:28.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:50:28.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:50:29.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:50:29.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:50:29.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:50:29.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:50:29.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:50:29.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:50:29.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:50:29.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-29T05:50:29.973 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:50:29.973 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-29T05:50:10.752853+0000 21'1 26:60 [1,0] 1 [1,0] 1 21'1 2024-03-29T05:50:10.565664+0000 21'1 2024-03-29T05:50:10.565664+0000 0 1 periodic scrub scheduled @ 2024-03-30T05:50:10.565664+0000 1 0 2024-03-29T05:50:29.973 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:50:29.973 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:50:29.974 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:50:29.991 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-29T05:50:29.991 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:50:29.991 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:50:29.991 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:50:29.992 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:50:29.992 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:50:29.992 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:50:29.992 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:50:30.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:50:30.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:50:30.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:50:30.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:50:30.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:50:30.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:50:30.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:50:30.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-29T05:50:32.449 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T05:50:32.449 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:50:32.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:50:32.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:50:32.994 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:50:32.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-29T05:50:32.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:50:32.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:50:32.996 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:50:32.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-29T05:50:32.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:50:32.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:50:32.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:50:32.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:50:32.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:50:33.040 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:33.042+0000 7f3063d317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:33.043 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:33.046+0000 7f3063d317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:33.044 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:33.050+0000 7f3063d317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:33.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:50:33.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:50:33.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:50:33.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:50:33.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:50:33.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:33.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:50:33.314 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:50:33.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:33.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:33.579 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:33.582+0000 7f3063d317c0 -1 Falling back to public interface 2024-03-29T05:50:33.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:34.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:34.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:34.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:50:34.677 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:50:34.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:34.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:34.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:36.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:36.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:50:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:50:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:36.063 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:36.066+0000 7f3063d317c0 -1 osd.0 26 log_to_monitors true 2024-03-29T05:50:36.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:37.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:37.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:37.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:50:37.330 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:50:37.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:37.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2866037527,v1:127.0.0.1:6803/2866037527] [v2:127.0.0.1:6804/2866037527,v1:127.0.0.1:6805/2866037527] exists,up e2025209-99bd-45ef-9487-8706ce2fe725 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:50:37.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:50:37.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:50:37.644 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:50:37.645 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:50:37.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:50:37.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:50:37.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:50:37.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:50:37.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:50:37.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:50:37.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:50:37.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:50:37.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:50:38.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:50:38.062 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:50:38.062 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:50:38.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:50:38.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:38.062 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:50:38.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-29T05:50:38.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-29T05:50:38.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-29T05:50:38.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:38.230 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:50:38.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-03-29T05:50:38.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-03-29T05:50:38.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672990' 2024-03-29T05:50:38.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:50:38.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:50:38.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-29T05:50:38.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-29T05:50:38.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672990 2-64424509460' 2024-03-29T05:50:38.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:38.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-29T05:50:38.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:38.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:50:38.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-29T05:50:38.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:38.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-29T05:50:38.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-29T05:50:38.514 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 128849018882 2024-03-29T05:50:38.514 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:50:38.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-29T05:50:38.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:50:39.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:50:39.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:50:40.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-29T05:50:40.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:40.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-03-29T05:50:40.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:40.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:50:40.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-03-29T05:50:40.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:40.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-03-29T05:50:40.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-03-29T05:50:40.123 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672990 2024-03-29T05:50:40.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:50:40.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672990 2024-03-29T05:50:40.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:50:40.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-29T05:50:40.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:50:40.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:50:40.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-29T05:50:40.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:50:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-29T05:50:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-29T05:50:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509460 2024-03-29T05:50:40.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:50:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-29T05:50:40.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:50:40.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:50:40.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:50:41.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:50:41.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:50:41.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:50:41.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:50:41.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:50:41.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:50:41.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:50:41.239 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:50:41.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:50:41.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:50:41.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:50:41.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:50:41.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:50:41.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:50:41.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:50:41.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T05:50:41.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T05:50:41.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-29T05:50:41.975 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:09.574+0000 7f2378841640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-03-29T05:50:41.976 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:09.574+0000 7f2378841640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-03-29T05:50:41.976 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:09.574+0000 7f2378841640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-29T05:50:41.976 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:10.558+0000 7f2378841640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:50:41.976 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:10.558+0000 7f2378841640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-29T05:50:41.976 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:10.558+0000 7f2378841640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-29T05:50:41.976 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:10.558+0000 7f2378841640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-29T05:50:41.976 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:50:10.558+0000 7f2378841640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:50:41.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-03-29T05:50:41.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:50:42.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:50:42.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:50:42.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:50:42.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:50:42.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:50:42.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:50:42.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:50:42.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:50:42.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:50:42.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:50:42.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:50:42.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:50:42.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:50:42.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:50:42.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:50:42.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:50:42.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:50:42.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:50:42.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:50:42.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:42.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:42.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:50:42.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:50:42.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:50:42.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:50:42.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:50:42.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:50:42.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:50:42.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:50:42.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:50:42.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:50:42.469 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:50:42.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:50:42.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:50:42.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:50:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:50:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:50:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:50:42.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:50:42.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:50:42.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:50:42.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:50:42.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:50:42.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:42.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:42.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:50:42.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:50:42.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:50:42.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:50:42.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:50:42.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:42.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:42.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:50:42.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:50:42.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:50:42.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:50:42.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:50:42.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:50:42.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T05:50:42.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:50:42.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:50:42.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:50:42.893 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:50:42.893 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:42.893 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:42.893 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:42.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T05:50:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:50:43.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:50:43.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:50:43.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:50:43.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:50:43.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:50:43.114 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:50:43.115 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:50:43.115 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:50:43.115 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:50:43.115 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:43.115 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:43.116 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:50:43.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:50:43.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:50:43.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:50:43.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:50:43.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:50:43.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:43.229 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:50:43.230 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:50:43.230 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:50:43.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-03-29T05:50:43.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:50:43.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:50:43.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:50:43.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:50:43.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:50:43.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:50:44.229 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:50:44.229 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:50:44.229 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:50:44.229 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:50:44.230 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:44.230 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:44.230 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:44.231 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:50:44.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:50:44.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-29T05:50:44.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:50:44.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:44.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-29T05:50:44.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:50:44.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:50:44.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71882717-faa0-4b60-a3b8-b2f5935247fe 2024-03-29T05:50:44.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 71882717-faa0-4b60-a3b8-b2f5935247fe' 2024-03-29T05:50:44.274 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 71882717-faa0-4b60-a3b8-b2f5935247fe 2024-03-29T05:50:44.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:50:44.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0VgZmD7+fERAAdVAUGEfPZLUKvFM8uIF+PA== 2024-03-29T05:50:44.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0VgZmD7+fERAAdVAUGEfPZLUKvFM8uIF+PA=="}' 2024-03-29T05:50:44.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71882717-faa0-4b60-a3b8-b2f5935247fe -i td/osd-scrub-repair/0/new.json 2024-03-29T05:50:44.548 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:50:44.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:50:44.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQC0VgZmD7+fERAAdVAUGEfPZLUKvFM8uIF+PA== --osd-uuid 71882717-faa0-4b60-a3b8-b2f5935247fe 2024-03-29T05:50:44.594 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:44.597+0000 7f4337ffc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:44.615 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:44.617+0000 7f4337ffc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:44.616 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:44.617+0000 7f4337ffc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:44.616 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:44.617+0000 7f4337ffc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:50:44.616 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:44.621+0000 7f4337ffc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:50:44.616 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:44.621+0000 7f4337ffc7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:50:44.707 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:44.709+0000 7f4337ffc7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:50:51.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:50:51.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:50:51.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:50:51.129 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:50:51.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:50:51.517 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:50:51.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:50:51.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:50:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:50:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:50:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:50:51.554 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:51.557+0000 7fb9513507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:51.586 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:51.589+0000 7fb9513507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:51.588 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:51.589+0000 7fb9513507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:51.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:50:51.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:50:51.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:50:51.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:50:51.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:50:51.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:51.853 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:50:51.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:50:51.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:51.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:52.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:52.145+0000 7fb9513507c0 -1 Falling back to public interface 2024-03-29T05:50:52.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:53.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:53.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:53.202 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:50:53.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:50:53.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:53.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:53.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:54.505 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:50:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:50:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:54.766 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:54.769+0000 7fb9513507c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:50:54.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:50:55.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:50:55.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:50:55.810 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:50:55.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:50:55.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:50:55.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:50:56.137 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3695896705,v1:127.0.0.1:6803/3695896705] [v2:127.0.0.1:6804/3695896705,v1:127.0.0.1:6805/3695896705] exists,up 71882717-faa0-4b60-a3b8-b2f5935247fe 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:50:56.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:50:56.142 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:50:56.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-29T05:50:56.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:50:56.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:50:56.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f21b7e84-576a-49ae-bfbb-e56f189e07ee 2024-03-29T05:50:56.144 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 f21b7e84-576a-49ae-bfbb-e56f189e07ee 2024-03-29T05:50:56.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f21b7e84-576a-49ae-bfbb-e56f189e07ee' 2024-03-29T05:50:56.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:50:56.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAVgZmlm7eCRAAzQzDS+cwgYEcZ5kVcCADPA== 2024-03-29T05:50:56.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAVgZmlm7eCRAAzQzDS+cwgYEcZ5kVcCADPA=="}' 2024-03-29T05:50:56.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f21b7e84-576a-49ae-bfbb-e56f189e07ee -i td/osd-scrub-repair/1/new.json 2024-03-29T05:50:56.579 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:50:56.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:50:56.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQDAVgZmlm7eCRAAzQzDS+cwgYEcZ5kVcCADPA== --osd-uuid f21b7e84-576a-49ae-bfbb-e56f189e07ee 2024-03-29T05:50:56.637 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:56.641+0000 7f07c4c7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:56.698 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:56.701+0000 7f07c4c7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:56.701+0000 7f07c4c7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:50:56.700 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:56.701+0000 7f07c4c7d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:50:56.700 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:56.701+0000 7f07c4c7d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:50:56.700 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:56.701+0000 7f07c4c7d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:50:56.774 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:50:56.777+0000 7f07c4c7d7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:51:03.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:51:03.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:51:03.523 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:51:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:51:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:51:03.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:51:03.912 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:51:03.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:51:03.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:51:03.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:51:03.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:51:03.946 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:03.949+0000 7f4efbd387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:03.985 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:03.989+0000 7f4efbd387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:03.987 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:03.989+0000 7f4efbd387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:51:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:51:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:51:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:51:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:51:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:04.253 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:51:04.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:51:04.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:51:04.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:04.543 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:04.545+0000 7f4efbd387c0 -1 Falling back to public interface 2024-03-29T05:51:04.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:05.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:05.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:05.570 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:51:05.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:51:05.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:05.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:51:05.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:06.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:06.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:06.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:51:06.874 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:51:06.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:06.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:51:07.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:07.216 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:07.221+0000 7f4efbd387c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:51:08.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:08.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:08.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:51:08.184 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:51:08.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:08.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:51:08.571 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2204096335,v1:127.0.0.1:6811/2204096335] [v2:127.0.0.1:6812/2204096335,v1:127.0.0.1:6813/2204096335] exists,up f21b7e84-576a-49ae-bfbb-e56f189e07ee 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:51:08.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:51:08.575 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:51:08.576 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:51:08.576 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:51:08.576 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:51:08.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:51:08.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3addf5da-b0eb-4945-a58f-545319e6153e 2024-03-29T05:51:08.579 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 3addf5da-b0eb-4945-a58f-545319e6153e 2024-03-29T05:51:08.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3addf5da-b0eb-4945-a58f-545319e6153e' 2024-03-29T05:51:08.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:51:08.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMVgZmefECJBAAcqutBvvboJaQPF+Y/mLJwA== 2024-03-29T05:51:08.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMVgZmefECJBAAcqutBvvboJaQPF+Y/mLJwA=="}' 2024-03-29T05:51:08.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3addf5da-b0eb-4945-a58f-545319e6153e -i td/osd-scrub-repair/2/new.json 2024-03-29T05:51:08.963 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:51:08.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T05:51:08.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQDMVgZmefECJBAAcqutBvvboJaQPF+Y/mLJwA== --osd-uuid 3addf5da-b0eb-4945-a58f-545319e6153e 2024-03-29T05:51:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:09.021+0000 7f332ba667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:09.077+0000 7f332ba667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:09.075 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:09.077+0000 7f332ba667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:09.075 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:09.077+0000 7f332ba667c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:51:09.075 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:09.077+0000 7f332ba667c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:51:09.075 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:09.077+0000 7f332ba667c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:51:09.156 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:09.157+0000 7f332ba667c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:51:16.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T05:51:16.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:51:16.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:51:16.244 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:51:16.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:51:16.638 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:51:16.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:51:16.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:51:16.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:51:16.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:51:16.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:51:16.674 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:16.677+0000 7f44959de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:16.697+0000 7f44959de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:16.697+0000 7f44959de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:16.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:51:16.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:51:16.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:51:16.993 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:51:16.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:51:16.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:51:16.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:16.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:51:16.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:16.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:51:17.263 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:17.265+0000 7f44959de7c0 -1 Falling back to public interface 2024-03-29T05:51:17.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:18.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:18.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:18.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:51:18.333 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:51:18.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:18.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:51:18.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:19.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:19.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:19.633 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:51:19.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:51:19.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:19.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:51:19.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:20.019 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:20.020+0000 7f44959de7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:51:20.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:20.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:20.935 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:51:20.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:51:20.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:20.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:51:21.317 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1888467627,v1:127.0.0.1:6819/1888467627] [v2:127.0.0.1:6820/1888467627,v1:127.0.0.1:6821/1888467627] exists,up 3addf5da-b0eb-4945-a58f-545319e6153e 2024-03-29T05:51:21.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:51:21.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:51:21.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:51:21.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-03-29T05:51:21.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-29T05:51:21.781 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'testpool' created 2024-03-29T05:51:21.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:51:22.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-03-29T05:51:23.415 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 2 2024-03-29T05:51:23.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-03-29T05:51:23.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:51:23.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:51:23.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:51:23.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:51:23.437 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:51:23.437 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:51:23.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:51:23.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:51:23.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:51:23.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:51:23.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:51:23.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:51:23.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:51:23.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:51:23.558 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:51:24.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:51:24.007 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:51:24.007 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:51:24.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:51:24.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:24.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:51:24.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-03-29T05:51:24.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-03-29T05:51:24.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-03-29T05:51:24.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:24.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:51:24.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-29T05:51:24.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-29T05:51:24.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-03-29T05:51:24.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:24.296 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:51:24.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-03-29T05:51:24.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-03-29T05:51:24.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2024-03-29T05:51:24.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:24.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-03-29T05:51:24.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:24.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:51:24.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-03-29T05:51:24.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:24.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-03-29T05:51:24.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-03-29T05:51:24.447 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836487 2024-03-29T05:51:24.447 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:51:24.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836487 2024-03-29T05:51:24.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:51:25.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:51:25.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:51:26.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-03-29T05:51:26.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:26.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:26.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-29T05:51:26.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:51:26.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-29T05:51:26.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:26.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-29T05:51:26.080 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672965 2024-03-29T05:51:26.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-29T05:51:26.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:51:26.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-29T05:51:26.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:26.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-03-29T05:51:26.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:26.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:51:26.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-03-29T05:51:26.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:26.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-03-29T05:51:26.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-03-29T05:51:26.430 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509442 2024-03-29T05:51:26.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:51:26.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-03-29T05:51:26.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:51:26.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:51:26.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:51:27.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:51:27.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:51:27.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:51:27.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:51:27.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:51:27.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:51:27.141 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:51:27.141 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:51:27.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:51:27.487 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:51:27.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:51:27.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:51:27.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:51:27.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:51:27.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:51:27.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-03-29T05:51:27.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-03-29T05:51:27.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T05:51:28.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-29T05:51:28.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:51:28.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:51:28.013 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:51:28.013 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:51:28.013 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:51:28.013 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:51:28.013 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:51:28.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:51:28.331 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:51:28.331 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:51:28.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:51:28.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:51:28.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:51:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:51:28.649 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:51:28.649 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:51:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:51:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:51:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:51:28.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:51:28.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T05:51:28.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:51:28.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:51:28.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:51:28.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:51:28.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:51:28.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-29T05:51:30.559 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:51:31.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:51:31.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:51:31.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:51:31.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:51:31.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:51:31.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:51:31.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:51:31.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-29T05:51:31.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:51:31.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:51:31.103 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:51:31.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:51:31.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:51:31.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:51:31.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:51:31.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:51:31.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:51:31.140 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:31.140+0000 7f2d413027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:31.142 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:31.144+0000 7f2d413027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:31.144 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:31.144+0000 7f2d413027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:51:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:51:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:51:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:51:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:51:31.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:31.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:51:31.430 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:51:31.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:31.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:31.694 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:31.696+0000 7f2d413027c0 -1 Falling back to public interface 2024-03-29T05:51:31.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:32.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:32.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:32.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:51:32.812 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:51:32.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:32.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:33.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:34.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:34.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:34.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:51:34.115 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:51:34.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:34.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:34.348 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:34.348+0000 7f2d413027c0 -1 osd.0 21 log_to_monitors true 2024-03-29T05:51:34.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:35.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:35.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:35.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:51:35.421 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:51:35.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:35.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:35.878 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1354334451,v1:127.0.0.1:6803/1354334451] [v2:127.0.0.1:6804/1354334451,v1:127.0.0.1:6805/1354334451] exists,up 71882717-faa0-4b60-a3b8-b2f5935247fe 2024-03-29T05:51:35.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:51:35.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:51:35.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:51:35.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:51:35.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:51:35.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:51:35.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:51:35.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:51:35.880 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:51:35.880 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:51:35.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:51:35.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:51:35.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:51:35.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:51:35.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:51:35.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:51:35.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:51:35.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:51:35.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:51:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:51:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:51:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:51:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:51:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:36.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:51:36.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-29T05:51:36.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-29T05:51:36.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-29T05:51:36.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:36.487 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:51:36.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-29T05:51:36.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-29T05:51:36.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2024-03-29T05:51:36.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:36.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:51:36.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-29T05:51:36.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-29T05:51:36.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509446' 2024-03-29T05:51:36.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:36.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-29T05:51:36.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:36.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:51:36.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-29T05:51:36.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:36.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-29T05:51:36.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-29T05:51:36.778 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 107374182402 2024-03-29T05:51:36.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:51:37.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-29T05:51:37.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:37.084 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-29T05:51:37.084 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:37.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:51:37.086 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-29T05:51:37.086 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-29T05:51:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-29T05:51:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672968 2024-03-29T05:51:37.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:51:37.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-29T05:51:37.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:37.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-29T05:51:37.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:37.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:51:37.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-29T05:51:37.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:37.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-29T05:51:37.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-29T05:51:37.438 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509446 2024-03-29T05:51:37.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:51:37.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509446 2024-03-29T05:51:37.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:51:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:51:38.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:51:39.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-29T05:51:39.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:51:39.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:51:39.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:51:39.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:51:39.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:51:39.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:51:39.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:51:39.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:51:39.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:51:39.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:51:39.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:51:39.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:51:39.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:51:39.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:51:39.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:51:40.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:51:40.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:51:40.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:51:40.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-03-29T05:51:40.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-29T05:51:40.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T05:51:40.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-29T05:51:40.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:51:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-03-29T05:51:40.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-03-29T05:51:40.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:51:40.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:51:40.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:51:40.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:51:40.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-03-29T05:51:40.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-03-29T05:51:40.836 INFO:tasks.workunit.client.0.smithi123.stdout:Affected PG 1.0 w/ primary 1 2024-03-29T05:51:40.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-03-29T05:51:40.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:51:40.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:51:40.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:51:40.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:51:41.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-03-29T05:51:21.685888+0000 2024-03-29T05:51:41.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-03-29T05:51:41.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-03-29T05:51:41.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-03-29T05:51:41.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-03-29T05:51:41.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T05:51:41.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:51:41.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:51:41.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T05:51:41.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-03-29T05:51:21.685888+0000 2024-03-29T05:51:41.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-03-29T05:51:41.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T05:51:41.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-29T05:51:41.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-29T05:51:41.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-03-29T05:51:41.465 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T05:51:41.465 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-29T05:51:41.465 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T05:51:41.465 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T05:51:41.465 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:51:41.465 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:51:41.465 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:51:41.466 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T05:51:41.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-03-29T05:51:41.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T05:51:41.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-03-29T05:51:41.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T05:51:41.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-29T05:51:41.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-03-29T05:51:41.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T05:51:41.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T05:51:41.581 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-29T05:51:41.581 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T05:51:41.582 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T05:51:41.582 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:51:41.582 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:51:41.582 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:51:41.583 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T05:51:41.583 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-03-29T05:51:41.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T05:51:41.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-03-29T05:51:41.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:51:41.703 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:51:41.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:51:41.994 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:51:41.994 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:51:41.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:51:41.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:41.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:51:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-03-29T05:51:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-03-29T05:51:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-03-29T05:51:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:42.172 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:51:42.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-29T05:51:42.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-29T05:51:42.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970' 2024-03-29T05:51:42.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:42.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:51:42.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-29T05:51:42.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-29T05:51:42.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672970 2-64424509448' 2024-03-29T05:51:42.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:42.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-03-29T05:51:42.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:42.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:51:42.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-03-29T05:51:42.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-03-29T05:51:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-03-29T05:51:42.473 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 107374182404 2024-03-29T05:51:42.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:51:42.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182403 -lt 107374182404 2024-03-29T05:51:42.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:51:43.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:51:43.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:51:44.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-03-29T05:51:44.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:44.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-29T05:51:44.072 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:44.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:51:44.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-29T05:51:44.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:44.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-29T05:51:44.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-29T05:51:44.076 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672970 2024-03-29T05:51:44.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:51:44.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-29T05:51:44.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:44.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-29T05:51:44.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:44.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:51:44.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-29T05:51:44.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:44.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-29T05:51:44.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-29T05:51:44.396 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509448 2024-03-29T05:51:44.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:51:44.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-29T05:51:44.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-03-29T05:51:44.720 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29 05:51:44.726620188+00:00 2024-03-29T05:51:44.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-03-29T05:51:44.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T05:51:44.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T05:51:44.722 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:51:44.722 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:51:44.722 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:51:44.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T05:51:44.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-03-29T05:51:44.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-03-29T05:51:44.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-29T05:51:44.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-29T05:51:44.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-29T05:51:44.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-29T05:51:44.980 INFO:tasks.workunit.client.0.smithi123.stdout:{"deep":true,"must":false,"stamp":"2024-03-15T05:50:04.825880+0000"}0 ) state now: "active+clean" 2024-03-29T05:51:44.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-03-29T05:51:44.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-29T05:51:44.981 INFO:tasks.workunit.client.0.smithi123.stdout:"active+clean" 2024-03-29T05:51:44.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-29T05:51:44.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-03-29T05:51:44.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-29T05:51:45.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-29T05:51:45.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-29T05:51:45.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-29T05:51:45.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-29T05:51:45.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-29T05:51:45.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-03-29T05:51:45.465 INFO:tasks.workunit.client.0.smithi123.stdout:1 ) state now: "active+clean" 2024-03-29T05:51:45.465 INFO:tasks.workunit.client.0.smithi123.stdout:"active+clean" 2024-03-29T05:51:45.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-29T05:51:45.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-29T05:51:45.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-03-29T05:51:45.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-29T05:51:45.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-29T05:51:45.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-29T05:51:45.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-29T05:51:45.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-29T05:51:45.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-29T05:51:45.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-03-29T05:51:45.920 INFO:tasks.workunit.client.0.smithi123.stdout:2 ) state now: "active+clean" 2024-03-29T05:51:45.920 INFO:tasks.workunit.client.0.smithi123.stdout:"active+clean" 2024-03-29T05:51:45.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-29T05:51:45.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-29T05:51:45.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-03-29T05:51:45.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-29T05:51:46.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-29T05:51:46.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-29T05:51:46.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-29T05:51:46.223 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-29T05:51:46.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-03-29T05:51:46.377 INFO:tasks.workunit.client.0.smithi123.stdout:3 ) state now: "active+clean+scrubbing+deep" 2024-03-29T05:51:46.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean+scrubbing+deep"' 2024-03-29T05:51:46.378 INFO:tasks.workunit.client.0.smithi123.stdout:found scrub 2024-03-29T05:51:46.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-29T05:51:46.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-03-29T05:51:46.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-03-29T05:51:46.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-03-29T05:51:46.380 INFO:tasks.workunit.client.0.smithi123.stdout:initiate_and_fetch_state ret: 0 2024-03-29T05:51:46.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-03-29T05:51:46.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-03-29T05:51:46.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T05:51:46.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-03-29T05:51:46.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-29T05:51:46.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-03-29T05:51:46.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T05:51:46.384 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-03-29T05:51:46.384 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-03-29T05:51:46.385 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-03-29T05:51:46.385 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:51:46.385 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:51:46.385 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:51:46.385 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.td/osd-scrub-repair.asok 2024-03-29T05:51:46.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-29T05:51:46.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-29T05:51:46.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-29T05:51:46.614 INFO:tasks.workunit.client.0.smithi123.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-29T05:51:46.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-29T05:51:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-29T05:51:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-03-29T05:51:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-29T05:51:46.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-29T05:51:46.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-29T05:51:46.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-29T05:51:46.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-29T05:51:47.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-29T05:51:47.072 INFO:tasks.workunit.client.0.smithi123.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-29T05:51:47.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-29T05:51:47.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-29T05:51:47.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-03-29T05:51:47.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-29T05:51:47.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-29T05:51:47.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-29T05:51:47.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-29T05:51:47.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-29T05:51:47.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-29T05:51:47.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-29T05:51:47.532 INFO:tasks.workunit.client.0.smithi123.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-29T05:51:47.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-29T05:51:47.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-03-29T05:51:47.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-29T05:51:47.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-29T05:51:47.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-29T05:51:47.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-29T05:51:47.834 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-29T05:51:47.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-29T05:51:47.992 INFO:tasks.workunit.client.0.smithi123.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-29T05:51:47.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-29T05:51:47.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-29T05:51:47.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-03-29T05:51:47.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-29T05:51:48.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-29T05:51:48.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-29T05:51:48.294 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-29T05:51:48.294 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-29T05:51:48.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-29T05:51:48.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-29T05:51:48.451 INFO:tasks.workunit.client.0.smithi123.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-29T05:51:48.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-29T05:51:48.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-03-29T05:51:48.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-03-29T05:51:48.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:51:48.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:51:48.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:51:48.761 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:51:48.761 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:51:48.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:51:48.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:48.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:51:48.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-03-29T05:51:48.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-03-29T05:51:48.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-03-29T05:51:48.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:48.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:51:49.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-29T05:51:49.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-29T05:51:49.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672973' 2024-03-29T05:51:49.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:51:49.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:51:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-29T05:51:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-29T05:51:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672973 2-64424509450' 2024-03-29T05:51:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:49.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-03-29T05:51:49.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:49.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:51:49.190 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-03-29T05:51:49.190 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:49.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-03-29T05:51:49.191 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 107374182406 2024-03-29T05:51:49.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-03-29T05:51:49.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:51:49.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-03-29T05:51:49.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:51:50.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:51:50.576 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:51:50.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182406 2024-03-29T05:51:50.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:50.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-29T05:51:50.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:50.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:51:50.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-29T05:51:50.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:50.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-29T05:51:50.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-29T05:51:50.908 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672973 2024-03-29T05:51:50.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:51:51.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-29T05:51:51.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:51:51.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-29T05:51:51.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:51:51.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:51:51.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-29T05:51:51.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:51:51.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-29T05:51:51.260 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509450 2024-03-29T05:51:51.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-29T05:51:51.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:51:51.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-29T05:51:51.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-29T05:51:51.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-29T05:51:51.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-29T05:51:51.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-29T05:51:51.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-29T05:51:52.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-03-29T05:51:52.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-03-29T05:51:52.065 INFO:tasks.workunit.client.0.smithi123.stdout:wait-scrub-end state now: "active+clean" 2024-03-29T05:51:52.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-29T05:51:52.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-03-29T05:51:52.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-29T05:51:52.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-03-29T05:51:52.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-03-29T05:51:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T05:51:52.346 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-03-29T05:51:46.364716+0000 21'1 26:48 [1,0] 1 [1,0] 1 0'0 2024-03-15T05:50:04.825880+0000 0'0 2024-03-15T05:50:04.825880+0000 0 0 deep scrubbing for 2s 0 0 2024-03-29T05:51:52.347 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T05:51:52.347 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T05:51:52.347 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T05:51:52.363 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-29T05:51:52.363 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:51:52.363 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:51:52.363 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:51:52.364 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:51:52.364 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:51:52.364 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:51:52.364 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:51:52.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:51:52.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:51:52.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:51:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:51:53.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:51:53.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:51:53.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:51:53.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:51:53.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:51:53.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:51:53.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-29T05:51:54.777 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T05:51:54.777 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T05:51:55.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:51:55.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:51:55.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:51:55.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:51:55.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:51:55.087 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:51:55.087 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:51:55.087 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:51:55.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:51:55.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-29T05:51:55.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:51:55.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:51:55.089 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:51:55.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:51:55.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:51:55.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:51:55.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:51:55.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:51:55.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:51:55.131 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:55.132+0000 7f03191157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:55.134 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:55.136+0000 7f03191157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:55.135 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:55.136+0000 7f03191157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:51:55.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:55.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:55.675 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:55.676+0000 7f03191157c0 -1 Falling back to public interface 2024-03-29T05:51:55.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:56.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:56.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:56.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:51:56.790 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:51:56.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:56.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:57.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:58.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:58.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:58.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:51:58.101 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:51:58.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:58.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:58.193 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:51:58.195+0000 7f03191157c0 -1 osd.0 26 log_to_monitors true 2024-03-29T05:51:58.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:51:59.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:51:59.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:51:59.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:51:59.448 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:51:59.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:51:59.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:51:59.841 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/345806941,v1:127.0.0.1:6803/345806941] [v2:127.0.0.1:6804/345806941,v1:127.0.0.1:6805/345806941] exists,up 71882717-faa0-4b60-a3b8-b2f5935247fe 2024-03-29T05:51:59.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:51:59.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:51:59.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:51:59.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:51:59.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:51:59.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:51:59.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:51:59.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:51:59.843 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:51:59.843 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:51:59.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:51:59.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:51:59.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:51:59.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:51:59.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:51:59.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:51:59.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:51:59.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:51:59.957 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:52:00.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:52:00.270 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:52:00.270 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:52:00.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:52:00.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:52:00.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:52:00.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-29T05:52:00.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-29T05:52:00.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-29T05:52:00.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:52:00.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:52:00.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-29T05:52:00.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-29T05:52:00.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672976' 2024-03-29T05:52:00.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:52:00.599 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:52:00.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-29T05:52:00.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-29T05:52:00.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672976 2-64424509453' 2024-03-29T05:52:00.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:52:00.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-29T05:52:00.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:52:00.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:52:00.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-29T05:52:00.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:52:00.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-29T05:52:00.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-29T05:52:00.745 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 128849018882 2024-03-29T05:52:00.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:52:01.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-29T05:52:01.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:52:01.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:52:01.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-29T05:52:01.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:52:01.050 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-29T05:52:01.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:52:01.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-29T05:52:01.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-29T05:52:01.052 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672976 2024-03-29T05:52:01.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:52:01.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-29T05:52:01.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:52:01.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-29T05:52:01.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:52:01.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:52:01.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-29T05:52:01.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:52:01.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-29T05:52:01.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-29T05:52:01.462 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509453 2024-03-29T05:52:01.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:52:01.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509453 2024-03-29T05:52:01.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:52:02.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:52:02.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:52:03.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-03-29T05:52:03.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:52:03.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:52:03.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:52:03.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:52:03.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:52:03.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:52:03.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:52:03.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:52:03.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:52:03.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:52:03.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:52:03.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:52:03.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:52:03.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:52:03.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:52:04.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:52:04.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:52:04.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:52:04.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-29T05:52:04.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T05:52:04.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:52:04.131 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-29T05:52:04.131 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T05:52:04.131 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:52:04.132 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:52:04.132 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:52:04.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T05:52:04.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-29T05:52:04.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T05:52:04.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:52:04.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T05:52:04.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:52:04.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T05:52:04.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:52:04.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:52:04.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:52:04.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:52:04.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:52:06.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:52:06.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:52:06.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:52:06.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:52:06.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T05:52:06.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:52:06.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:52:06.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:52:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:52:07.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:52:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:52:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:52:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:52:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:52:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:52:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-29T05:52:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:52:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T05:52:07.721 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T05:52:07.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-29T05:52:07.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T05:52:07.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T05:52:07.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:52:07.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:52:07.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:52:07.763 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:07.763+0000 7fda7798f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:07.765 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:07.767+0000 7fda7798f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:07.766 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:07.767+0000 7fda7798f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:52:08.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:08.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:08.299+0000 7fda7798f7c0 -1 Falling back to public interface 2024-03-29T05:52:08.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:09.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:09.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:09.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:52:09.400 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:52:09.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:09.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:09.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:10.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:10.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:10.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:52:10.711 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:52:10.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:10.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:10.843 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:10.843+0000 7fda7798f7c0 -1 osd.0 31 log_to_monitors true 2024-03-29T05:52:11.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:12.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:12.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:12.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:52:12.167 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:52:12.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:12.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:12.478 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/1697716006,v1:127.0.0.1:6803/1697716006] [v2:127.0.0.1:6804/1697716006,v1:127.0.0.1:6805/1697716006] exists,up 71882717-faa0-4b60-a3b8-b2f5935247fe 2024-03-29T05:52:12.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:52:12.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:52:12.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:52:12.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:52:12.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:52:12.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:52:12.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:52:12.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:52:12.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:52:12.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:52:12.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:52:12.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:52:12.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:52:12.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:52:12.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:52:12.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:52:12.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:52:12.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:52:12.585 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:52:12.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:52:12.937 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:52:12.937 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:52:12.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:52:12.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:52:12.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:52:13.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-29T05:52:13.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-29T05:52:13.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-29T05:52:13.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:52:13.084 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:52:13.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-29T05:52:13.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-29T05:52:13.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672979' 2024-03-29T05:52:13.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:52:13.273 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:52:13.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-29T05:52:13.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-29T05:52:13.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672979 2-64424509457' 2024-03-29T05:52:13.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:52:13.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-29T05:52:13.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:52:13.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:52:13.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-29T05:52:13.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:52:13.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-29T05:52:13.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-29T05:52:13.407 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 150323855362 2024-03-29T05:52:13.407 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:52:13.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-29T05:52:13.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:52:14.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:52:14.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:52:15.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-29T05:52:15.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:52:15.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:52:15.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-29T05:52:15.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:52:15.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-29T05:52:15.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:52:15.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-29T05:52:15.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-29T05:52:15.011 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672979 2024-03-29T05:52:15.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:52:15.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672979 2024-03-29T05:52:15.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:52:15.294 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-29T05:52:15.294 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:52:15.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:52:15.297 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-29T05:52:15.297 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:52:15.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-29T05:52:15.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-29T05:52:15.298 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509457 2024-03-29T05:52:15.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:52:15.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-29T05:52:15.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:52:15.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:52:15.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:52:15.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T05:52:15.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:52:15.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:52:15.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:52:15.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:52:15.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:52:15.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:52:15.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:52:16.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T05:52:16.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:52:16.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:52:16.309 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:52:16.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T05:52:16.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:52:16.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:52:16.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T05:52:16.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-29T05:52:16.686 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:51:49.380+0000 7f4ee24c7640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-29T05:52:16.686 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:51:49.380+0000 7f4ee24c7640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-29T05:52:16.686 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:51:49.380+0000 7f4ee24c7640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-29T05:52:16.686 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:51:49.380+0000 7f4ee24c7640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-29T05:52:16.686 INFO:tasks.workunit.client.0.smithi123.stdout:2024-03-29T05:51:49.380+0000 7f4ee24c7640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:52:16.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:52:16.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:52:16.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:52:16.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:52:16.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:52:16.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:52:16.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:52:16.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:52:16.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:52:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:52:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:52:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:52:16.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:52:16.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:52:16.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:52:16.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:52:16.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:52:16.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:16.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:16.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:52:16.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:52:16.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:52:16.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:52:16.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:52:16.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:52:16.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:52:16.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:52:16.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:52:16.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:52:16.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:52:16.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:52:16.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:52:16.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:52:16.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:52:16.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:52:16.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:52:16.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:52:16.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:52:16.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:52:16.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:52:16.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:52:16.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:16.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:16.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:52:16.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:52:16.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:52:16.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:52:16.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:52:16.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:16.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:16.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:52:16.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:52:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:52:16.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:52:16.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T05:52:17.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:52:17.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:52:17.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:52:17.200 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:52:17.200 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:17.200 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:17.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:17.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T05:52:17.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:52:17.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:52:17.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:52:17.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:52:17.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:52:17.413 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:52:17.413 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:52:17.413 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:52:17.413 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:52:17.414 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:52:17.414 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:17.414 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:17.414 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:52:17.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:52:17.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:52:17.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:52:17.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:52:17.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:52:17.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:52:17.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:52:17.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:52:17.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-29T05:52:17.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:52:17.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:52:17.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:52:17.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:52:17.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:52:17.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:52:18.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:52:18.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:52:18.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:52:18.527 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:52:18.527 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:18.527 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:18.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:18.528 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:52:18.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:52:18.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:52:18.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-29T05:52:18.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-29T05:52:18.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:52:18.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:52:18.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:52:18.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:52:18.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:52:18.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=02d985e8-18f6-4820-820b-6427d2a903ee 2024-03-29T05:52:18.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 02d985e8-18f6-4820-820b-6427d2a903ee' 2024-03-29T05:52:18.570 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 02d985e8-18f6-4820-820b-6427d2a903ee 2024-03-29T05:52:18.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:52:18.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASVwZmszw8IxAAgfRX6D4jlBPq9dY+JINcIQ== 2024-03-29T05:52:18.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASVwZmszw8IxAAgfRX6D4jlBPq9dY+JINcIQ=="}' 2024-03-29T05:52:18.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 02d985e8-18f6-4820-820b-6427d2a903ee -i td/osd-scrub-repair/0/new.json 2024-03-29T05:52:18.829 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:52:18.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:52:18.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQASVwZmszw8IxAAgfRX6D4jlBPq9dY+JINcIQ== --osd-uuid 02d985e8-18f6-4820-820b-6427d2a903ee 2024-03-29T05:52:18.883 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:18.883+0000 7f8d7ccbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:18.886 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:18.887+0000 7f8d7ccbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:18.887 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:18.887+0000 7f8d7ccbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:18.888 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:18.887+0000 7f8d7ccbd7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:52:18.888 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:18.887+0000 7f8d7ccbd7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:52:18.888 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:18.887+0000 7f8d7ccbd7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:52:18.980 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:18.979+0000 7f8d7ccbd7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:52:25.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:52:25.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:52:25.488 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:52:25.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:52:25.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:52:25.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:52:25.868 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:52:25.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:52:25.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:52:25.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:52:25.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:52:25.903 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:25.903+0000 7f672725f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:25.935 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:25.935+0000 7f672725f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:25.936 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:25.939+0000 7f672725f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:26.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:52:26.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:52:26.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:52:26.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:52:26.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:52:26.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:26.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:52:26.197 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:52:26.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:26.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:26.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:26.506 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:26.507+0000 7f672725f7c0 -1 Falling back to public interface 2024-03-29T05:52:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:52:27.494 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:52:27.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:27.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:27.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:28.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:28.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:28.810 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:52:28.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:52:28.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:28.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:29.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:29.155 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:29.155+0000 7f672725f7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:52:30.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:30.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:30.120 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:52:30.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:52:30.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:30.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:30.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:31.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:31.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:31.451 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:52:31.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:52:31.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:31.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:31.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:32.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:32.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:32.800 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:52:32.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:52:32.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:32.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:33.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:34.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:34.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:34.318 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:52:34.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:52:34.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:34.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:34.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:35.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:35.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:35.626 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:52:35.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:52:35.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:35.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:36.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:37.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:37.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:37.063 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:52:37.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:52:37.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:37.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:37.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:38.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:38.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:38.370 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:52:38.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:52:38.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:38.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:38.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:39.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:39.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:39.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:52:39.690 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:52:39.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:39.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:40.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:41.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:41.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:41.077 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:52:41.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:52:41.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:41.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:41.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:42.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:42.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:42.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:52:42.404 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:52:42.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:42.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:42.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:43.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:43.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:43.726 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:52:43.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:52:43.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:43.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:44.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:45.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:45.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:45.386 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:52:45.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:52:45.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:45.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:46.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:47.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:47.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:47.226 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:52:47.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:52:47.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:47.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:47.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:48.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:48.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:48.944 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:52:48.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:52:48.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:48.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:49.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:50.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:50.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:50.865 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:52:50.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:52:50.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:50.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:51.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:52.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:52.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:52.741 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:52:52.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:52:52.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:52.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:53.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:52:54.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:52:54.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:52:54.694 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:52:54.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:52:54.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:52:54.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:52:55.006 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/243999055,v1:127.0.0.1:6803/243999055] [v2:127.0.0.1:6804/243999055,v1:127.0.0.1:6805/243999055] exists,up 02d985e8-18f6-4820-820b-6427d2a903ee 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:52:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:52:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:52:55.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:52:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:52:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:52:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:52:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=401956f6-edb1-4e20-b196-1916448ec7bf 2024-03-29T05:52:55.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 401956f6-edb1-4e20-b196-1916448ec7bf' 2024-03-29T05:52:55.014 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 401956f6-edb1-4e20-b196-1916448ec7bf 2024-03-29T05:52:55.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:52:55.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3VwZmvgP/ARAA+1dM0sUb4agKjkXqXQ/zNw== 2024-03-29T05:52:55.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3VwZmvgP/ARAA+1dM0sUb4agKjkXqXQ/zNw=="}' 2024-03-29T05:52:55.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 401956f6-edb1-4e20-b196-1916448ec7bf -i td/osd-scrub-repair/1/new.json 2024-03-29T05:52:55.464 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:52:55.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:52:55.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA3VwZmvgP/ARAA+1dM0sUb4agKjkXqXQ/zNw== --osd-uuid 401956f6-edb1-4e20-b196-1916448ec7bf 2024-03-29T05:52:55.526 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:55.530+0000 7fb65f8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:55.575 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:55.578+0000 7fb65f8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:55.577 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:55.582+0000 7fb65f8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:52:55.577 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:55.582+0000 7fb65f8b47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:52:55.577 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:55.582+0000 7fb65f8b47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:52:55.577 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:55.582+0000 7fb65f8b47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:52:55.647 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:52:55.650+0000 7fb65f8b47c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:53:02.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:53:02.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:53:02.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:53:02.709 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:53:02.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:53:03.094 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:53:03.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:53:03.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:53:03.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:53:03.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:53:03.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:53:03.130 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:03.134+0000 7fe68a4df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:03.132 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:03.138+0000 7fe68a4df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:03.133 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:03.138+0000 7fe68a4df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:03.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:53:03.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:53:03.393 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:53:03.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:53:03.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:53:03.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:53:03.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:03.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:53:03.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:03.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:03.687 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:03.690+0000 7fe68a4df7c0 -1 Falling back to public interface 2024-03-29T05:53:03.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:04.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:04.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:04.768 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:53:04.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:53:04.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:04.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:05.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:53:06.081 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:53:06.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:06.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:06.274 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:06.278+0000 7fe68a4df7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:53:06.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:07.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:07.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:07.404 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:53:07.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:53:07.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:07.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:07.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:08.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:08.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:08.753 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:53:08.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:53:08.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:08.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:09.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:10.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:10.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:10.074 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:53:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:53:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:10.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:11.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:11.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:11.456 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:53:11.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:53:11.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:11.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:11.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:12.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:12.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:12.780 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:53:12.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:53:12.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:12.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:13.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:14.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:14.099 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:53:14.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:53:14.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:14.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:14.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:15.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:15.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:53:15.416 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:53:15.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:15.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:15.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:16.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:53:16.786 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:53:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:17.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:18.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:18.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:18.365 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:53:18.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:53:18.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:18.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:18.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:19.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:19.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:19.725 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:53:19.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:53:19.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:19.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:20.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:21.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:21.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:21.421 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:53:21.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:53:21.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:21.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:21.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:22.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:22.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:22.844 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:53:22.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:53:22.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:22.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:23.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:24.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:24.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:24.303 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:53:24.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:53:24.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:24.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:24.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:25.778 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:53:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:53:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:26.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:27.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:27.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:27.575 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:53:27.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:53:27.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:27.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:28.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:29.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:29.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:29.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:53:29.463 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:53:29.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:29.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:30.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:31.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:31.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:31.159 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:53:31.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:53:31.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:31.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:31.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:32.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:32.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:32.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:53:32.726 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:53:32.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:32.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:33.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:34.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:34.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:34.135 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T05:53:34.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T05:53:34.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:34.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:53:34.458 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3270120869,v1:127.0.0.1:6811/3270120869] [v2:127.0.0.1:6812/3270120869,v1:127.0.0.1:6813/3270120869] exists,up 401956f6-edb1-4e20-b196-1916448ec7bf 2024-03-29T05:53:34.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:53:34.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:53:34.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:53:34.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T05:53:34.465 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:53:34.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b69a1d25-6529-466e-94a6-191b8958ea64 2024-03-29T05:53:34.465 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 b69a1d25-6529-466e-94a6-191b8958ea64 2024-03-29T05:53:34.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b69a1d25-6529-466e-94a6-191b8958ea64' 2024-03-29T05:53:34.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:53:34.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeVwZmI9Y+HRAAiezorLR/HFrjh/Gs6QACHQ== 2024-03-29T05:53:34.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeVwZmI9Y+HRAAiezorLR/HFrjh/Gs6QACHQ=="}' 2024-03-29T05:53:34.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b69a1d25-6529-466e-94a6-191b8958ea64 -i td/osd-scrub-repair/2/new.json 2024-03-29T05:53:34.902 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:53:34.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T05:53:34.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBeVwZmI9Y+HRAAiezorLR/HFrjh/Gs6QACHQ== --osd-uuid b69a1d25-6529-466e-94a6-191b8958ea64 2024-03-29T05:53:34.962 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:34.965+0000 7f2fc8b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:35.013 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:35.017+0000 7f2fc8b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:35.014 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:35.017+0000 7f2fc8b177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:35.015 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:35.017+0000 7f2fc8b177c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:53:35.015 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:35.017+0000 7f2fc8b177c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:53:35.015 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:35.017+0000 7f2fc8b177c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:53:35.091 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:35.093+0000 7f2fc8b177c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:53:42.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T05:53:42.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:53:42.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:53:42.513 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:53:42.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:53:42.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:53:42.889 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:53:42.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:53:42.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:53:42.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:53:42.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:53:42.925 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:42.929+0000 7fb1e815a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:42.954 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:42.957+0000 7fb1e815a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:42.955 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:42.957+0000 7fb1e815a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:53:43.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:53:43.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:53:43.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:53:43.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:53:43.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:53:43.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:43.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:53:43.220 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:53:43.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:43.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:43.502 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:43.505+0000 7fb1e815a7c0 -1 Falling back to public interface 2024-03-29T05:53:43.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:44.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:44.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:44.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:53:44.549 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:53:44.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:44.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:44.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:45.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:45.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:45.851 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:53:45.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:53:45.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:45.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:46.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:46.351 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:53:46.352+0000 7fb1e815a7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:53:47.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:47.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:47.159 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:53:47.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:53:47.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:47.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:47.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:48.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:48.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:48.587 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:53:48.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:53:48.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:48.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:48.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:49.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:49.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:49.908 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:53:49.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:53:49.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:49.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:50.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:51.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:51.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:51.284 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:53:51.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:53:51.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:51.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:52.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:53.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:53.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:53.042 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:53:53.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:53:53.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:53.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:53.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:54.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:54.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:54.614 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:53:54.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:53:54.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:54.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:54.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:55.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:55.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:55.936 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:53:55.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:53:55.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:55.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:56.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:57.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:57.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:57.367 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:53:57.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:53:57.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:57.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:57.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:53:58.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:53:58.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:53:58.690 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:53:58.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:53:58.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:53:58.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:53:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:00.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:00.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:00.075 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:54:00.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:54:00.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:00.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:00.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:01.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:01.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:01.490 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:54:01.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:54:01.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:01.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:01.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:02.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:02.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:02.887 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:54:02.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:54:02.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:02.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:03.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:04.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:04.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:04.567 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:54:04.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:54:04.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:04.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:05.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:06.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:06.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:06.139 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:54:06.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:54:06.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:06.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:07.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:08.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:08.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:08.074 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:54:08.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:54:08.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:08.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:08.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:09.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:09.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:09.592 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:54:09.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:54:09.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:09.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:09.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:10.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:10.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:10.941 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:54:10.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:54:10.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:10.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:11.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:12.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:12.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:12.805 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:54:12.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:54:12.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:12.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:13.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:14.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:14.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:14.887 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T05:54:14.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T05:54:14.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:14.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:54:15.279 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3768572025,v1:127.0.0.1:6819/3768572025] [v2:127.0.0.1:6820/3768572025,v1:127.0.0.1:6821/3768572025] exists,up b69a1d25-6529-466e-94a6-191b8958ea64 2024-03-29T05:54:15.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:54:15.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:54:15.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:54:15.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-29T05:54:15.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T05:54:15.666 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T05:54:15.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T05:54:15.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T05:54:16.148 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T05:54:16.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:54:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T05:54:19.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-29T05:54:19.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:54:19.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:54:19.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:54:19.612 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:54:19.614 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:54:19.614 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:54:19.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:54:19.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:54:19.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:54:19.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:54:19.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:54:19.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:54:19.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:54:19.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:54:19.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:54:20.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:54:20.158 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:54:20.158 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:54:20.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:54:20.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:20.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:54:20.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-29T05:54:20.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-29T05:54:20.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-29T05:54:20.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:20.326 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:54:20.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444045 2024-03-29T05:54:20.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444045 2024-03-29T05:54:20.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-73014444045' 2024-03-29T05:54:20.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:20.467 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:54:20.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789957 2024-03-29T05:54:20.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789957 2024-03-29T05:54:20.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-73014444045 2-158913789957' 2024-03-29T05:54:20.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:20.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-29T05:54:20.608 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:20.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:54:20.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-29T05:54:20.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:20.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-29T05:54:20.610 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836499 2024-03-29T05:54:20.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-29T05:54:20.611 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:20.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836499 2024-03-29T05:54:20.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:54:21.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:54:21.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:22.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-29T05:54:22.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:22.231 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-73014444045 2024-03-29T05:54:22.231 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:22.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:54:22.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-73014444045 2024-03-29T05:54:22.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:22.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444045 2024-03-29T05:54:22.235 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 73014444045 2024-03-29T05:54:22.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 73014444045' 2024-03-29T05:54:22.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:54:22.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444045 -lt 73014444045 2024-03-29T05:54:22.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:22.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:22.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789957 2024-03-29T05:54:22.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:54:22.584 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789957 2024-03-29T05:54:22.584 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:22.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789957 2024-03-29T05:54:22.586 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 158913789957 2024-03-29T05:54:22.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789957' 2024-03-29T05:54:22.587 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:54:22.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789958 -lt 158913789957 2024-03-29T05:54:22.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:54:22.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:22.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:23.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T05:54:23.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:54:23.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:54:23.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:54:23.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:54:23.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:54:23.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:54:23.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:54:23.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:54:23.586 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:54:23.586 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:23.586 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:23.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T05:54:23.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-29T05:54:24.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T05:54:24.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T05:54:25.798 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T05:54:25.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:54:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-29T05:54:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T05:54:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:54:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:54:26.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:54:26.822 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:54:26.822 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:54:26.823 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:54:26.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:54:26.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:54:26.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:54:26.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:54:26.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:54:26.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:54:26.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:54:26.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:54:26.950 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:54:27.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:54:27.315 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:54:27.315 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:54:27.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:54:27.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:27.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:54:27.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-29T05:54:27.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-29T05:54:27.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-29T05:54:27.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:27.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:54:27.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444047 2024-03-29T05:54:27.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444047 2024-03-29T05:54:27.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-73014444047' 2024-03-29T05:54:27.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:27.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:54:27.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789960 2024-03-29T05:54:27.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789960 2024-03-29T05:54:27.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-73014444047 2-158913789960' 2024-03-29T05:54:27.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:27.802 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-29T05:54:27.802 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:27.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:54:27.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-29T05:54:27.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:27.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-29T05:54:27.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-29T05:54:27.805 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836501 2024-03-29T05:54:27.805 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:28.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-29T05:54:28.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:54:29.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:54:29.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:29.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-29T05:54:29.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:29.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-73014444047 2024-03-29T05:54:29.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:29.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:54:29.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-73014444047 2024-03-29T05:54:29.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:29.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444047 2024-03-29T05:54:29.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 73014444047' 2024-03-29T05:54:29.432 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 73014444047 2024-03-29T05:54:29.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:54:29.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444048 -lt 73014444047 2024-03-29T05:54:29.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:29.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789960 2024-03-29T05:54:29.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:29.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:54:29.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789960 2024-03-29T05:54:29.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:29.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789960 2024-03-29T05:54:29.757 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 158913789960 2024-03-29T05:54:29.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789960' 2024-03-29T05:54:29.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:54:30.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789960 -lt 158913789960 2024-03-29T05:54:30.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:54:30.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:30.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:30.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:54:30.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:54:30.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:54:30.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:54:30.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:54:30.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:54:30.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:54:30.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:54:30.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:54:30.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:54:30.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:30.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:31.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:54:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:54:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:54:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T05:54:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-29T05:54:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-29T05:54:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T05:54:31.447 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-29T05:54:31.447 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-29T05:54:31.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:54:31.448 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-29T05:54:31.448 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T05:54:31.448 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:54:31.448 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:54:31.448 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:54:31.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-29T05:54:31.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:54:31.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-29T05:54:32.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T05:54:32.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:54:32.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:54:32.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:54:32.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:54:32.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:54:32.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:54:32.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:54:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T05:54:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:54:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:54:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:54:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:54:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:54:32.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T05:54:34.374 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-29T05:54:34.375 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-29T05:54:34.375 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T05:54:34.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:54:34.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:54:34.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:54:34.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:54:34.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:54:34.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:54:34.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:54:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:54:34.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:54:34.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:54:34.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:54:34.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:54:34.919 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:54:34.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:54:34.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:54:34.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:54:34.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:54:34.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:54:34.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:54:34.957 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:34.955+0000 7f4bc1bec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:54:34.959 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:34.959+0000 7f4bc1bec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:54:34.961 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:34.959+0000 7f4bc1bec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:54:35.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:54:35.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:54:35.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:54:35.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:54:35.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:54:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:54:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:54:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:54:35.496 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:35.495+0000 7f4bc1bec7c0 -1 Falling back to public interface 2024-03-29T05:54:35.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:36.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:36.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:36.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:54:36.568 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:54:36.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:36.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:54:36.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:37.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:37.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:37.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:54:37.892 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:54:37.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:37.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:54:38.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:54:38.384 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:38.382+0000 7f4bc1bec7c0 -1 osd.1 58 log_to_monitors true 2024-03-29T05:54:39.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:54:39.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:39.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:54:39.198 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:54:39.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:39.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:54:39.619 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 64 up_thru 64 down_at 59 last_clean_interval [17,58) [v2:127.0.0.1:6810/3051774165,v1:127.0.0.1:6811/3051774165] [v2:127.0.0.1:6812/3051774165,v1:127.0.0.1:6813/3051774165] exists,up 401956f6-edb1-4e20-b196-1916448ec7bf 2024-03-29T05:54:39.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:54:39.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:54:39.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:54:39.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:54:39.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:54:39.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:54:39.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:54:39.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:54:39.621 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:54:39.621 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:54:39.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:54:39.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:54:39.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:54:39.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:54:39.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:54:39.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:54:39.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:54:39.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:54:39.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:54:40.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:54:40.183 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:54:40.183 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:54:40.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:54:40.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:40.184 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:54:40.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-29T05:54:40.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-29T05:54:40.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-29T05:54:40.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:40.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:54:40.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906946 2024-03-29T05:54:40.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906946 2024-03-29T05:54:40.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-274877906946' 2024-03-29T05:54:40.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:40.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:54:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789963 2024-03-29T05:54:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789963 2024-03-29T05:54:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-274877906946 2-158913789963' 2024-03-29T05:54:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:40.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-29T05:54:40.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:40.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:54:40.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-29T05:54:40.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:40.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-29T05:54:40.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-29T05:54:40.747 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836505 2024-03-29T05:54:40.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:41.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836505 2024-03-29T05:54:41.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:54:42.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:54:42.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:42.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-03-29T05:54:42.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:42.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-274877906946 2024-03-29T05:54:42.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:42.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:54:42.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-274877906946 2024-03-29T05:54:42.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:42.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906946 2024-03-29T05:54:42.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 274877906946' 2024-03-29T05:54:42.387 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 274877906946 2024-03-29T05:54:42.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:54:42.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906946 -lt 274877906946 2024-03-29T05:54:42.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:42.688 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789963 2024-03-29T05:54:42.688 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:42.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:54:42.691 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789963 2024-03-29T05:54:42.691 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:42.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789963 2024-03-29T05:54:42.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789963' 2024-03-29T05:54:42.692 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 158913789963 2024-03-29T05:54:42.692 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:54:43.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789964 -lt 158913789963 2024-03-29T05:54:43.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:54:43.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:43.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:43.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:54:43.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:54:43.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:54:43.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:54:43.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:54:43.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:54:43.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:54:43.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:54:43.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:54:43.811 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:54:43.811 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:43.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:44.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T05:54:44.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T05:54:44.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T05:54:44.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T05:54:44.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:54:44.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T05:54:44.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:54:44.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:54:44.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:54:44.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:54:44.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:54:44.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:54:44.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:54:44.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:54:44.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:54:44.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:54:44.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:44.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T05:54:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T05:54:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T05:54:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T05:54:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T05:54:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T05:54:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T05:54:45.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T05:54:45.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T05:54:45.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T05:54:45.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T05:54:45.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T05:54:45.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:54:45.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T05:54:45.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:54:45.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:54:45.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:54:45.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:54:45.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:54:45.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:54:45.629 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:54:45.629 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:54:45.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:54:45.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:54:45.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:45.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:46.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:54:46.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:54:46.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:54:46.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-29T05:54:46.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T05:54:46.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T05:54:46.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:54:46.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:54:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-29T05:54:46.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-29T05:54:46.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:54:46.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:54:46.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:54:46.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:54:47.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-29T05:54:42.300906+0000 2024-03-29T05:54:47.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T05:54:47.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:54:42.300906+0000 2024-03-29T05:54:47.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:54:47.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:54:47.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:54:47.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:54:47.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:54:47.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:54:47.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:54:47.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:54:47.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:54:42.300906+0000 '>' 2024-03-29T05:54:42.300906+0000 2024-03-29T05:54:47.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:54:48.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:54:48.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:54:48.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:54:48.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:54:48.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:54:48.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:54:48.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:54:48.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:54:42.300906+0000 '>' 2024-03-29T05:54:42.300906+0000 2024-03-29T05:54:48.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:54:49.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:54:49.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:54:49.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:54:49.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:54:49.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:54:49.703 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:54:49.703 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:54:50.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:54:42.300906+0000 '>' 2024-03-29T05:54:42.300906+0000 2024-03-29T05:54:50.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:54:51.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:54:51.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:54:51.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:54:51.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:54:51.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:54:51.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:54:51.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:54:51.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:54:48.280645+0000 '>' 2024-03-29T05:54:42.300906+0000 2024-03-29T05:54:51.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:54:51.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-29T05:54:51.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:54:51.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:54:51.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:54:51.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:54:51.342 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:54:51.342 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:54:51.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:54:51.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:54:51.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:54:51.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:54:51.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:54:51.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:54:51.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:54:51.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:54:51.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:54:51.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:54:51.756 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:54:51.756 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:54:51.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:54:51.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:51.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:54:51.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-29T05:54:51.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-29T05:54:51.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-29T05:54:51.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:51.925 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:54:52.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906949 2024-03-29T05:54:52.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906949 2024-03-29T05:54:52.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-274877906949' 2024-03-29T05:54:52.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:54:52.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:54:52.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789967 2024-03-29T05:54:52.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789967 2024-03-29T05:54:52.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-274877906949 2-158913789967' 2024-03-29T05:54:52.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:52.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-29T05:54:52.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:52.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:54:52.205 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-29T05:54:52.205 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:52.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-29T05:54:52.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-29T05:54:52.206 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836508 2024-03-29T05:54:52.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:52.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-29T05:54:52.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:54:53.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:54:53.532 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:54:53.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-29T05:54:53.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:53.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-274877906949 2024-03-29T05:54:53.855 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:53.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:54:53.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-274877906949 2024-03-29T05:54:53.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:53.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906949 2024-03-29T05:54:53.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 274877906949' 2024-03-29T05:54:53.859 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 274877906949 2024-03-29T05:54:53.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:54:54.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906949 -lt 274877906949 2024-03-29T05:54:54.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:54:54.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789967 2024-03-29T05:54:54.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:54:54.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:54:54.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789967 2024-03-29T05:54:54.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:54:54.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789967 2024-03-29T05:54:54.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789967' 2024-03-29T05:54:54.199 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 158913789967 2024-03-29T05:54:54.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:54:54.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789967 -lt 158913789967 2024-03-29T05:54:54.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:54:54.545 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:54.545 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:54.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:54:54.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:54:54.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:54:54.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:54:54.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:54:54.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:54:54.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:54:54.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:54:55.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:54:55.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:54:55.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:54:55.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:54:55.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:54:55.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:54:55.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:54:55.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-29T05:54:55.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-29T05:54:55.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:54:55.657 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-29T05:54:55.657 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T05:54:55.657 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:54:55.658 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:54:55.658 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:54:56.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-29T05:54:56.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-29T05:54:56.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:54:56.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:54:56.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:54:56.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:54:56.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:54:56.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:54:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T05:54:59.202 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-29T05:54:59.202 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-29T05:54:59.202 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T05:54:59.203 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T05:54:59.203 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T05:54:59.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:54:59.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:54:59.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:54:59.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:54:59.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:54:59.489 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:54:59.490 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:54:59.491 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:54:59.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:54:59.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:54:59.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:54:59.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:54:59.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:54:59.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:54:59.529 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:59.530+0000 7ff7ea2807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:54:59.531 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:59.534+0000 7ff7ea2807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:54:59.533 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:54:59.534+0000 7ff7ea2807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:54:59.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:54:59.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:54:59.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:54:59.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:54:59.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:54:59.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:54:59.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:54:59.812 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:54:59.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:54:59.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:55:00.100 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:00.102+0000 7ff7ea2807c0 -1 Falling back to public interface 2024-03-29T05:55:00.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:01.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:01.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:01.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:55:01.129 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:55:01.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:01.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:55:01.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:02.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:02.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:02.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:55:02.453 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:55:02.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:02.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:55:02.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:03.062 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:03.066+0000 7ff7ea2807c0 -1 osd.1 76 log_to_monitors true 2024-03-29T05:55:03.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:03.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:03.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:55:03.949 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:55:03.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:03.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:55:04.334 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 77 last_clean_interval [64,76) [v2:127.0.0.1:6810/2043024472,v1:127.0.0.1:6811/2043024472] [v2:127.0.0.1:6812/2043024472,v1:127.0.0.1:6813/2043024472] exists,up 401956f6-edb1-4e20-b196-1916448ec7bf 2024-03-29T05:55:04.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:55:04.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:55:04.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:55:04.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:55:04.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:55:04.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:55:04.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:55:04.336 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:55:04.336 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:55:04.336 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:55:04.336 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:55:04.336 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:55:04.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:55:04.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:55:04.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:55:04.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:55:04.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:55:04.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:55:04.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:55:04.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:55:04.834 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:55:04.834 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:55:04.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:55:04.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:55:04.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:55:05.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-03-29T05:55:05.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-03-29T05:55:05.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-03-29T05:55:05.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:55:05.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:55:05.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-03-29T05:55:05.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-03-29T05:55:05.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-356482285570' 2024-03-29T05:55:05.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:55:05.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:55:05.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789970 2024-03-29T05:55:05.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789970 2024-03-29T05:55:05.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-356482285570 2-158913789970' 2024-03-29T05:55:05.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:55:05.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-03-29T05:55:05.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:55:05.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:55:05.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:55:05.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-03-29T05:55:05.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-03-29T05:55:05.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-03-29T05:55:05.418 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836512 2024-03-29T05:55:05.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:55:05.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-03-29T05:55:05.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:55:05.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:55:05.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285570 2024-03-29T05:55:05.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:55:05.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285570 2024-03-29T05:55:05.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:55:05.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-03-29T05:55:05.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-03-29T05:55:05.762 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 356482285570 2024-03-29T05:55:05.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:55:06.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 356482285570 2024-03-29T05:55:06.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:55:07.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:55:07.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:55:07.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-03-29T05:55:07.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:55:07.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-158913789970 2024-03-29T05:55:07.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:55:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:55:07.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-158913789970 2024-03-29T05:55:07.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:55:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789970 2024-03-29T05:55:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 158913789970' 2024-03-29T05:55:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 158913789970 2024-03-29T05:55:07.789 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:55:08.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789971 -lt 158913789970 2024-03-29T05:55:08.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:55:08.125 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:55:08.125 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:55:08.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:55:08.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:55:08.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:55:08.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:55:08.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:55:08.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:55:08.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:55:08.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:55:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:55:08.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:55:08.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:55:08.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:55:09.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T05:55:09.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T05:55:09.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T05:55:09.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T05:55:09.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:55:09.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T05:55:09.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:55:09.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:55:09.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:55:09.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:55:09.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:55:09.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:55:09.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:55:09.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:55:09.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:55:09.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:55:09.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:55:09.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:55:09.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T05:55:09.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T05:55:09.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T05:55:09.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T05:55:09.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T05:55:09.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T05:55:09.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T05:55:09.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T05:55:09.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T05:55:10.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T05:55:10.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T05:55:10.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T05:55:10.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T05:55:10.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T05:55:10.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T05:55:10.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:55:10.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:55:10.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:55:10.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:55:10.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:55:10.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:55:10.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:55:10.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:55:10.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:55:10.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:55:10.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:55:11.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:55:11.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:55:11.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:55:11.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T05:55:11.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T05:55:11.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:55:11.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:55:11.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:55:11.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:55:11.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:55:11.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:55:11.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:55:11.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:55:11.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:55:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:55:11.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:55:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:55:11.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:55:11.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:55:11.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:55:11.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:55:11.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:55:11.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:55:11.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:55:11.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:55:11.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:55:11.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:55:11.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:55:11.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:55:11.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:11.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:11.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:55:11.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:55:11.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:55:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:55:11.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:55:11.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:55:11.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:55:11.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:55:11.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:55:11.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:55:11.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:55:11.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:55:11.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:55:11.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:55:11.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:55:11.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:55:11.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:55:11.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:55:11.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:55:11.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:11.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:11.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:55:11.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:55:11.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:55:11.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:55:11.464 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:55:11.464 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:11.464 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:11.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-29T05:55:11.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-29T05:55:11.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:55:11.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:55:11.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:55:11.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:55:11.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:55:11.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T05:55:11.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:55:11.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:55:11.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:55:11.865 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:55:11.866 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:11.866 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:11.866 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:55:11.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T05:55:12.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:55:12.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:55:12.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:55:12.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:55:12.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:55:12.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:55:12.077 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:55:12.077 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:55:12.077 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:55:12.077 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:55:12.078 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:12.078 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:12.078 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:55:12.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:55:12.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:55:12.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:55:12.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:55:12.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:55:12.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:55:12.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:55:12.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:55:12.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-29T05:55:12.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:55:12.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:55:12.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:55:12.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:55:12.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:55:12.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:55:13.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:55:13.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:55:13.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:55:13.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:55:13.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:13.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:13.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:55:13.177 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:55:13.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:55:13.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:55:13.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:55:13.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:55:13.218 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:55:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:55:13.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:55:13.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:55:13.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=95983e88-443c-49e1-8157-84e7111416bc 2024-03-29T05:55:13.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 95983e88-443c-49e1-8157-84e7111416bc' 2024-03-29T05:55:13.221 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 95983e88-443c-49e1-8157-84e7111416bc 2024-03-29T05:55:13.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:55:13.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBVwZmRKlnDhAAbOyy3fMKXus+R7y9yJosHQ== 2024-03-29T05:55:13.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBVwZmRKlnDhAAbOyy3fMKXus+R7y9yJosHQ=="}' 2024-03-29T05:55:13.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 95983e88-443c-49e1-8157-84e7111416bc -i td/osd-scrub-repair/0/new.json 2024-03-29T05:55:13.496 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:55:13.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:55:13.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDBVwZmRKlnDhAAbOyy3fMKXus+R7y9yJosHQ== --osd-uuid 95983e88-443c-49e1-8157-84e7111416bc 2024-03-29T05:55:13.547 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:13.549+0000 7fd318f2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:13.563 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:13.565+0000 7fd318f2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:13.564 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:13.565+0000 7fd318f2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:13.565 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:13.565+0000 7fd318f2e7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:55:13.565 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:13.565+0000 7fd318f2e7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:55:13.565 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:13.565+0000 7fd318f2e7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:55:13.663 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:13.665+0000 7fd318f2e7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:55:20.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:55:20.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:55:20.175 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:55:20.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:55:20.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:55:20.570 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:55:20.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:55:20.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:55:20.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:55:20.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:55:20.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:55:20.601 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:20.601+0000 7f6772f5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:20.603 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:20.605+0000 7f6772f5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:20.604 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:20.605+0000 7f6772f5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:20.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:55:20.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:21.185 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:21.185+0000 7f6772f5e7c0 -1 Falling back to public interface 2024-03-29T05:55:21.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:22.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:22.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:22.256 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:55:22.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:55:22.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:22.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:22.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:23.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:23.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:23.573 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:55:23.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:55:23.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:23.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:23.847 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:23.849+0000 7f6772f5e7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:55:23.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:24.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:24.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:24.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:55:24.891 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:55:24.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:24.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:25.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:26.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:26.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:26.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:55:26.217 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:55:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:26.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:26.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:27.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:27.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:27.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:55:27.541 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:55:27.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:27.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:27.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:28.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:28.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:28.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:55:28.937 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:55:28.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:28.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:29.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:30.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:30.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:30.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:55:30.261 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:55:30.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:30.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:30.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:31.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:31.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:31.591 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:55:31.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:55:31.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:31.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:31.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:32.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:32.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:32.951 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:55:32.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:55:32.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:32.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:33.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:34.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:34.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:34.270 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:55:34.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:55:34.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:34.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:34.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:35.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:35.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:35.784 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:55:35.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:55:35.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:35.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:36.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:37.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:37.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:37.095 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:55:37.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:55:37.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:37.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:37.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:38.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:38.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:38.409 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:55:38.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:55:38.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:38.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:38.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:39.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:39.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:39.966 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:55:39.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:55:39.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:39.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:40.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:41.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:41.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:41.835 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:55:41.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:55:41.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:41.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:42.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:43.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:43.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:43.737 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:55:43.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:55:43.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:43.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:44.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:45.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:45.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:45.600 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:55:45.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:55:45.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:45.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:46.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:47.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:47.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:47.258 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:55:47.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:55:47.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:47.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:48.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:49.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:49.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:49.037 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:55:49.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:55:49.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:49.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:49.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:55:50.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:55:50.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:50.353 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:55:50.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:55:50.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:50.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:55:50.669 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2858262960,v1:127.0.0.1:6803/2858262960] [v2:127.0.0.1:6804/2858262960,v1:127.0.0.1:6805/2858262960] exists,up 95983e88-443c-49e1-8157-84e7111416bc 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:55:50.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:55:50.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:55:50.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:55:50.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:55:50.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:55:50.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2899e72-592d-48a4-b089-438fe37f67ea 2024-03-29T05:55:50.675 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 a2899e72-592d-48a4-b089-438fe37f67ea 2024-03-29T05:55:50.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a2899e72-592d-48a4-b089-438fe37f67ea' 2024-03-29T05:55:50.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:55:50.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmVwZm5VGyKRAArDWYxGX1rcsrWxKVC4/mWw== 2024-03-29T05:55:50.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmVwZm5VGyKRAArDWYxGX1rcsrWxKVC4/mWw=="}' 2024-03-29T05:55:50.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2899e72-592d-48a4-b089-438fe37f67ea -i td/osd-scrub-repair/1/new.json 2024-03-29T05:55:51.075 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:55:51.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:55:51.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDmVwZm5VGyKRAArDWYxGX1rcsrWxKVC4/mWw== --osd-uuid a2899e72-592d-48a4-b089-438fe37f67ea 2024-03-29T05:55:51.132 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:51.132+0000 7fd247c0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:51.170 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:51.168+0000 7fd247c0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:51.170 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:51.168+0000 7fd247c0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:51.170 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:51.168+0000 7fd247c0b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:55:51.170 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:51.168+0000 7fd247c0b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:55:51.170 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:51.168+0000 7fd247c0b7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:55:51.252 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:51.252+0000 7fd247c0b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:55:58.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:55:58.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:55:58.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:55:58.498 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:55:58.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:55:58.874 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:55:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:55:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:55:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:55:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:55:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:55:58.910 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:58.908+0000 7f5a7a9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:58.941 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:58.940+0000 7f5a7a9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:58.942 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:58.944+0000 7f5a7a9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:55:59.212 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:55:59.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:55:59.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:55:59.493 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:55:59.492+0000 7f5a7a9ff7c0 -1 Falling back to public interface 2024-03-29T05:55:59.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:00.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:00.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:00.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:56:00.554 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:56:00.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:00.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:00.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:01.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:01.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:01.872 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:56:01.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:56:01.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:01.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:02.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:02.246 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:02.244+0000 7f5a7a9ff7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:56:03.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:03.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:03.172 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:56:03.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:56:03.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:03.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:03.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:04.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:04.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:04.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:56:04.478 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:56:04.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:04.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:04.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:05.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:05.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:05.913 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:56:05.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:56:05.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:05.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:07.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:07.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:07.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:56:07.265 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:56:07.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:07.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:07.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:08.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:08.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:08.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:56:08.584 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:56:08.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:08.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:09.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:10.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:10.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:10.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:56:10.034 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:56:10.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:10.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:10.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:11.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:11.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:11.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:56:11.477 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:56:11.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:11.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:11.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:12.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:12.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:12.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:56:12.791 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:56:12.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:12.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:13.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:14.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:14.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:14.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:56:14.303 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:56:14.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:14.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:14.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:15.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:15.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:15.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:56:15.732 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:56:15.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:15.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:16.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:17.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:17.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:17.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:56:17.045 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:56:17.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:17.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:17.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:18.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:18.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:18.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:56:18.621 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:56:18.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:18.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:19.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:20.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:20.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:20.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:56:20.263 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:56:20.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:20.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:21.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:22.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:22.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:22.169 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:56:22.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:56:22.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:22.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:23.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:24.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:24.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:24.182 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:56:24.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:56:24.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:24.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:25.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:26.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:26.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:26.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:56:26.022 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:56:26.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:26.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:26.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:27.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:27.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:27.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:56:27.767 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:56:27.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:27.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:28.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:29.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:29.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:29.570 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:56:29.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:56:29.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:29.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:56:29.880 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1147739266,v1:127.0.0.1:6811/1147739266] [v2:127.0.0.1:6812/1147739266,v1:127.0.0.1:6813/1147739266] exists,up a2899e72-592d-48a4-b089-438fe37f67ea 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:56:29.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:56:29.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:56:29.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=94003743-94e9-4cf7-baa5-12013e584b6a 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 94003743-94e9-4cf7-baa5-12013e584b6a' 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 94003743-94e9-4cf7-baa5-12013e584b6a 2024-03-29T05:56:29.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:56:29.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANWAZmFx1ENhAASbiAqdP2k8SRyE6NmEtObA== 2024-03-29T05:56:29.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANWAZmFx1ENhAASbiAqdP2k8SRyE6NmEtObA=="}' 2024-03-29T05:56:29.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 94003743-94e9-4cf7-baa5-12013e584b6a -i td/osd-scrub-repair/2/new.json 2024-03-29T05:56:30.377 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:56:30.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T05:56:30.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQANWAZmFx1ENhAASbiAqdP2k8SRyE6NmEtObA== --osd-uuid 94003743-94e9-4cf7-baa5-12013e584b6a 2024-03-29T05:56:30.436 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:30.435+0000 7fabb21057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:56:30.479 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:30.479+0000 7fabb21057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:56:30.480 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:30.479+0000 7fabb21057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:56:30.481 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:30.479+0000 7fabb21057c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:56:30.481 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:30.479+0000 7fabb21057c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:56:30.481 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:30.479+0000 7fabb21057c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:56:30.558 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:30.555+0000 7fabb21057c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:56:38.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T05:56:38.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:56:38.020 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:56:38.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:56:38.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:56:38.415 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:56:38.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:56:38.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:56:38.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:56:38.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:56:38.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:56:38.452 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:38.451+0000 7fb3fc2457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:56:38.454 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:38.451+0000 7fb3fc2457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:56:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:38.455+0000 7fb3fc2457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:56:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:56:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:56:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:56:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:56:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:56:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:38.748 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:56:38.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:56:38.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:38.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:38.989 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:38.987+0000 7fb3fc2457c0 -1 Falling back to public interface 2024-03-29T05:56:39.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:40.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:40.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:40.105 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:56:40.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:56:40.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:40.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:40.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:41.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:41.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:41.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:56:41.422 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:56:41.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:41.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:41.634 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:56:41.630+0000 7fb3fc2457c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:56:41.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:42.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:42.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:42.748 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:56:42.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:56:42.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:42.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:43.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:44.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:44.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:44.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:56:44.070 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:56:44.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:44.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:44.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:45.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:45.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:45.478 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:56:45.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:56:45.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:45.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:45.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:46.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:46.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:46.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:56:46.863 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:56:46.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:46.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:47.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:48.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:48.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:48.177 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:56:48.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:56:48.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:48.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:48.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:49.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:49.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:49.610 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:56:49.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:56:49.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:49.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:49.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:50.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:50.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:50.978 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:56:50.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:56:50.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:50.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:51.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:52.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:52.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:52.296 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:56:52.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:56:52.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:52.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:52.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:53.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:53.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:53.619 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:56:53.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:56:53.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:53.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:53.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:54.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:54.929 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:56:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:56:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:54.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:55.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:56.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:56.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:56.235 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:56:56.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:56:56.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:56.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:56.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:57.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:57.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:57.559 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:56:57.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:56:57.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:57.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:58.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:56:59.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:56:59.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:56:59.058 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:56:59.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:56:59.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:56:59.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:56:59.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:00.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:00.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:00.577 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:57:00.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:57:00.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:00.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:57:00.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:01.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:01.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:01.937 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:57:01.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:57:01.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:01.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:57:02.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:03.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:03.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:03.642 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:57:03.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:57:03.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:03.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:57:04.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:05.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:05.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:05.245 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:57:05.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:57:05.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:05.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:57:05.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:06.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:06.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:06.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:57:06.771 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:57:06.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:06.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:57:07.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:08.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:08.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:08.189 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T05:57:08.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T05:57:08.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:08.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:57:08.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:09.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:09.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:09.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T05:57:09.540 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T05:57:09.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:09.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:57:09.915 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4192769404,v1:127.0.0.1:6819/4192769404] [v2:127.0.0.1:6820/4192769404,v1:127.0.0.1:6821/4192769404] exists,up 94003743-94e9-4cf7-baa5-12013e584b6a 2024-03-29T05:57:09.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:57:09.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:57:09.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:57:09.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-29T05:57:09.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T05:57:10.409 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T05:57:10.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T05:57:10.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T05:57:11.002 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T05:57:11.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:57:12.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T05:57:14.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-29T05:57:14.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:57:14.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:57:14.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:57:14.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:57:14.834 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:57:14.834 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:57:14.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:57:14.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:57:14.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:57:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:57:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:57:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:57:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:57:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:57:14.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:57:15.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:57:15.289 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:57:15.289 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:57:15.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:57:15.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:15.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:57:15.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-29T05:57:15.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-29T05:57:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-29T05:57:15.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:15.465 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:57:15.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444046 2024-03-29T05:57:15.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444046 2024-03-29T05:57:15.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-73014444046' 2024-03-29T05:57:15.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:15.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:57:15.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822664 2024-03-29T05:57:15.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822664 2024-03-29T05:57:15.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-73014444046 2-154618822664' 2024-03-29T05:57:15.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:15.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-29T05:57:15.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:15.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:57:15.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-29T05:57:15.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:15.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-29T05:57:15.754 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836502 2024-03-29T05:57:15.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-29T05:57:15.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:57:16.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-29T05:57:16.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:16.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-73014444046 2024-03-29T05:57:16.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:16.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:57:16.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-73014444046 2024-03-29T05:57:16.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:16.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444046 2024-03-29T05:57:16.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 73014444046' 2024-03-29T05:57:16.091 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 73014444046 2024-03-29T05:57:16.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:57:16.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444046 -lt 73014444046 2024-03-29T05:57:16.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:16.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822664 2024-03-29T05:57:16.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:16.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:57:16.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822664 2024-03-29T05:57:16.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:16.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822664 2024-03-29T05:57:16.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822664' 2024-03-29T05:57:16.462 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 154618822664 2024-03-29T05:57:16.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:57:16.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822663 -lt 154618822664 2024-03-29T05:57:16.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:57:17.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:57:17.773 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:57:18.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822664 -lt 154618822664 2024-03-29T05:57:18.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:57:18.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:18.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:18.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T05:57:18.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:57:18.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:57:18.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:57:18.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:57:18.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:57:18.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:57:18.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:57:18.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T05:57:18.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:57:18.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:18.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T05:57:19.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-29T05:57:19.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T05:57:19.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T05:57:20.615 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T05:57:20.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T05:57:21.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-29T05:57:21.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-29T05:57:22.339 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 2 allow_ec_overwrites to true 2024-03-29T05:57:22.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T05:57:22.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:57:22.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:57:22.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:57:22.359 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:57:22.360 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:57:22.360 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:57:22.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:57:22.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:57:22.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:57:22.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:57:22.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:57:22.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:57:22.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:57:22.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:57:22.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:57:22.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:57:22.790 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:57:22.790 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:57:22.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:57:22.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:22.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:57:22.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-29T05:57:22.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-29T05:57:22.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-29T05:57:22.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:22.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:57:23.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=73014444048 2024-03-29T05:57:23.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 73014444048 2024-03-29T05:57:23.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-73014444048' 2024-03-29T05:57:23.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:23.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:57:23.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822666 2024-03-29T05:57:23.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822666 2024-03-29T05:57:23.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-73014444048 2-154618822666' 2024-03-29T05:57:23.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:23.223 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-29T05:57:23.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:23.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:57:23.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-29T05:57:23.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:23.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-29T05:57:23.226 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836504 2024-03-29T05:57:23.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-29T05:57:23.227 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:57:23.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-03-29T05:57:23.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:57:24.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:57:24.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:57:24.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-29T05:57:24.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:24.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:24.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-73014444048 2024-03-29T05:57:24.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:57:24.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-73014444048 2024-03-29T05:57:24.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:24.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=73014444048 2024-03-29T05:57:24.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 73014444048' 2024-03-29T05:57:24.902 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 73014444048 2024-03-29T05:57:24.902 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:57:25.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 73014444048 -lt 73014444048 2024-03-29T05:57:25.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:25.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822666 2024-03-29T05:57:25.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:25.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:57:25.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822666 2024-03-29T05:57:25.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:25.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822666 2024-03-29T05:57:25.314 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 154618822666 2024-03-29T05:57:25.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822666' 2024-03-29T05:57:25.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:57:25.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822666 -lt 154618822666 2024-03-29T05:57:25.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:57:25.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:25.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:26.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:57:26.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:57:26.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:57:26.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:57:26.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:57:26.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:57:26.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:57:26.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:57:26.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:57:26.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:57:26.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:26.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:26.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:57:26.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:57:26.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:57:26.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T05:57:26.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-29T05:57:26.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-29T05:57:26.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T05:57:27.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-29T05:57:27.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-29T05:57:27.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:57:27.140 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-29T05:57:27.140 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T05:57:27.140 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:57:27.140 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:57:27.141 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:57:27.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-29T05:57:27.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:57:27.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-29T05:57:27.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T05:57:27.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:57:27.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:57:27.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:57:27.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:57:27.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T05:57:27.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:57:27.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:57:27.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:57:27.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:57:27.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:57:27.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:57:27.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:57:27.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:57:27.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:57:27.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:57:28.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:57:28.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T05:57:28.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:57:28.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:57:28.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:57:28.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:57:28.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:57:28.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T05:57:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-29T05:57:30.555 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-29T05:57:30.555 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T05:57:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:57:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:57:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:57:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:57:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:57:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:57:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:57:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:57:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:57:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:57:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:57:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:57:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:57:31.093 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:57:31.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:57:31.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:57:31.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:57:31.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:57:31.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:57:31.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:57:31.129 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:31.129+0000 7f8021eed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:57:31.132 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:31.133+0000 7f8021eed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:57:31.133 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:31.133+0000 7f8021eed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:57:31.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:57:31.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:57:31.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:57:31.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:57:31.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:57:31.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:31.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:57:31.414 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:57:31.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:31.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:31.673 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:31.673+0000 7f8021eed7c0 -1 Falling back to public interface 2024-03-29T05:57:31.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:32.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:32.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:32.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:57:32.758 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:57:32.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:32.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:33.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:34.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:34.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:34.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:57:34.120 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:57:34.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:34.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:34.455 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:34.457+0000 7f8021eed7c0 -1 osd.1 61 log_to_monitors true 2024-03-29T05:57:34.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:35.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:35.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:35.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:57:35.476 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:57:35.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:35.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 69 up_thru 69 down_at 63 last_clean_interval [17,61) [v2:127.0.0.1:6810/1440809178,v1:127.0.0.1:6811/1440809178] [v2:127.0.0.1:6812/1440809178,v1:127.0.0.1:6813/1440809178] exists,up a2899e72-592d-48a4-b089-438fe37f67ea 2024-03-29T05:57:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:57:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:57:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:57:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:57:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:57:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:57:35.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:57:35.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:57:35.984 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:57:35.984 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:57:35.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:57:35.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:57:35.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:57:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:57:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:57:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:57:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:57:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:57:36.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:57:36.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:57:36.499 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:57:36.499 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:57:36.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:57:36.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:36.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:57:36.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-29T05:57:36.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-29T05:57:36.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-29T05:57:36.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:36.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:57:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-03-29T05:57:36.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-03-29T05:57:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-296352743426' 2024-03-29T05:57:36.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:36.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:57:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822670 2024-03-29T05:57:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822670 2024-03-29T05:57:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-296352743426 2-154618822670' 2024-03-29T05:57:37.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:37.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-29T05:57:37.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:37.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:57:37.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-29T05:57:37.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-29T05:57:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-29T05:57:37.079 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836508 2024-03-29T05:57:37.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:57:37.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-29T05:57:37.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:57:38.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:57:38.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:57:38.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-29T05:57:38.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:38.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-296352743426 2024-03-29T05:57:38.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:38.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:57:38.700 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-296352743426 2024-03-29T05:57:38.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:38.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-03-29T05:57:38.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 296352743426' 2024-03-29T05:57:38.702 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 296352743426 2024-03-29T05:57:38.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:57:38.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743426 -lt 296352743426 2024-03-29T05:57:38.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:38.998 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822670 2024-03-29T05:57:38.998 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:39.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:57:39.000 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822670 2024-03-29T05:57:39.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:39.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822670 2024-03-29T05:57:39.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822670' 2024-03-29T05:57:39.002 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 154618822670 2024-03-29T05:57:39.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:57:39.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822670 -lt 154618822670 2024-03-29T05:57:39.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:57:39.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:39.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:39.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:57:39.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:57:39.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:57:39.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:57:39.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:57:39.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:57:39.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:57:39.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:57:40.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:57:40.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:57:40.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:40.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:40.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:57:40.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:57:40.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:57:40.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-29T05:57:40.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T05:57:40.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T05:57:40.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T05:57:40.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:57:41.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-29T05:57:41.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-29T05:57:41.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:57:41.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:57:41.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:57:41.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:57:41.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-29T05:57:20.466791+0000 2024-03-29T05:57:41.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T05:57:41.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T05:57:20.466791+0000 2024-03-29T05:57:41.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T05:57:41.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T05:57:41.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:57:41.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:57:41.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:57:41.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:57:41.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:57:41.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:57:41.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:57:20.466791+0000 '>' 2024-03-29T05:57:20.466791+0000 2024-03-29T05:57:41.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:57:42.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:57:42.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:57:42.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:57:42.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:57:42.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:57:42.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:57:42.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:57:42.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:57:20.466791+0000 '>' 2024-03-29T05:57:20.466791+0000 2024-03-29T05:57:42.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:57:43.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:57:43.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:57:43.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:57:43.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:57:43.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:57:43.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:57:43.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:57:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:57:20.466791+0000 '>' 2024-03-29T05:57:20.466791+0000 2024-03-29T05:57:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T05:57:45.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T05:57:45.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T05:57:45.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T05:57:45.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T05:57:45.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T05:57:45.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T05:57:45.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T05:57:45.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T05:57:39.695383+0000 '>' 2024-03-29T05:57:20.466791+0000 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:57:45.595 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:57:45.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:57:45.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:57:45.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:57:45.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:57:45.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:57:45.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:57:45.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:57:45.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:57:45.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:57:46.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:57:46.042 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:57:46.042 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:57:46.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:57:46.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:46.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:57:46.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-03-29T05:57:46.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-03-29T05:57:46.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-03-29T05:57:46.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:46.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:57:46.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743429 2024-03-29T05:57:46.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743429 2024-03-29T05:57:46.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-296352743429' 2024-03-29T05:57:46.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:46.328 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:57:46.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822673 2024-03-29T05:57:46.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822673 2024-03-29T05:57:46.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-296352743429 2-154618822673' 2024-03-29T05:57:46.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:46.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:46.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-03-29T05:57:46.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:57:46.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-03-29T05:57:46.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:46.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-03-29T05:57:46.503 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836511 2024-03-29T05:57:46.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-03-29T05:57:46.504 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:57:46.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-03-29T05:57:46.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:57:47.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:57:47.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:57:48.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-03-29T05:57:48.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:48.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-296352743429 2024-03-29T05:57:48.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:48.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:57:48.141 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-296352743429 2024-03-29T05:57:48.141 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:48.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743429 2024-03-29T05:57:48.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 296352743429' 2024-03-29T05:57:48.143 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 296352743429 2024-03-29T05:57:48.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:57:48.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743429 -lt 296352743429 2024-03-29T05:57:48.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:57:48.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822673 2024-03-29T05:57:48.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:57:48.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:57:48.482 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822673 2024-03-29T05:57:48.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:57:48.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822673 2024-03-29T05:57:48.484 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 154618822673 2024-03-29T05:57:48.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822673' 2024-03-29T05:57:48.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:57:48.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822673 -lt 154618822673 2024-03-29T05:57:48.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:57:48.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:48.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:49.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:57:49.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:57:49.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:57:49.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:57:49.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:57:49.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:57:49.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:57:49.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:57:49.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:57:49.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:57:49.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:57:49.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:57:49.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:57:49.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:57:49.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:57:49.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-29T05:57:49.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-29T05:57:49.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T05:57:49.907 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-29T05:57:49.907 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T05:57:49.907 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T05:57:49.907 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:57:49.908 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T05:57:50.223 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-29T05:57:50.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-29T05:57:50.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T05:57:50.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T05:57:50.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:57:50.637 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:57:50.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:57:50.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:57:50.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:57:51.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:57:51.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T05:57:51.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T05:57:51.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T05:57:51.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T05:57:51.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T05:57:51.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:57:51.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T05:57:53.864 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-29T05:57:53.864 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-29T05:57:53.864 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T05:57:53.864 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T05:57:53.865 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T05:57:54.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:57:54.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T05:57:54.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T05:57:54.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T05:57:54.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T05:57:54.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:57:54.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:57:54.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:57:54.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T05:57:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-29T05:57:54.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T05:57:54.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T05:57:54.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T05:57:54.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T05:57:54.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T05:57:54.205 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:54.204+0000 7f3be467a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:57:54.207 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:54.208+0000 7f3be467a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:57:54.209 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:54.208+0000 7f3be467a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T05:57:54.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:54.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:54.742 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:54.740+0000 7f3be467a7c0 -1 Falling back to public interface 2024-03-29T05:57:54.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:57:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:57:55.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:55.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:56.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:57.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:57.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:57.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:57:57.197 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T05:57:57.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:57.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:57.374 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:57:57.372+0000 7f3be467a7c0 -1 osd.1 80 log_to_monitors true 2024-03-29T05:57:57.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:57:58.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:57:58.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:57:58.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:57:58.505 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T05:57:58.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:57:58.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 87 up_thru 87 down_at 81 last_clean_interval [69,80) [v2:127.0.0.1:6810/3818365691,v1:127.0.0.1:6811/3818365691] [v2:127.0.0.1:6812/3818365691,v1:127.0.0.1:6813/3818365691] exists,up a2899e72-592d-48a4-b089-438fe37f67ea 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T05:57:58.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T05:57:58.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T05:57:58.910 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T05:57:58.910 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T05:57:58.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T05:57:58.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T05:57:58.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T05:57:59.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T05:57:59.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T05:57:59.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T05:57:59.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T05:57:59.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T05:57:59.017 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T05:57:59.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T05:57:59.411 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T05:57:59.411 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T05:57:59.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T05:57:59.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:59.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T05:57:59.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-29T05:57:59.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-29T05:57:59.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-29T05:57:59.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:59.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T05:57:59.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154754 2024-03-29T05:57:59.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154754 2024-03-29T05:57:59.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-373662154754' 2024-03-29T05:57:59.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T05:57:59.901 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T05:58:00.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822677 2024-03-29T05:58:00.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822677 2024-03-29T05:58:00.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-373662154754 2-154618822677' 2024-03-29T05:58:00.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:58:00.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-29T05:58:00.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:58:00.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T05:58:00.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-29T05:58:00.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:58:00.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-29T05:58:00.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-29T05:58:00.055 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836514 2024-03-29T05:58:00.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T05:58:00.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-03-29T05:58:00.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:58:00.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154754 2024-03-29T05:58:00.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:58:00.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T05:58:00.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154754 2024-03-29T05:58:00.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:58:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154754 2024-03-29T05:58:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2024-03-29T05:58:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 373662154754 2024-03-29T05:58:00.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:58:00.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 373662154754 2024-03-29T05:58:00.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T05:58:01.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T05:58:01.789 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T05:58:02.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154754 -lt 373662154754 2024-03-29T05:58:02.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T05:58:02.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-154618822677 2024-03-29T05:58:02.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T05:58:02.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T05:58:02.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-154618822677 2024-03-29T05:58:02.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T05:58:02.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822677 2024-03-29T05:58:02.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 154618822677' 2024-03-29T05:58:02.114 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 154618822677 2024-03-29T05:58:02.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T05:58:02.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822677 -lt 154618822677 2024-03-29T05:58:02.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T05:58:02.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:58:02.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:58:02.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T05:58:02.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T05:58:02.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T05:58:02.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T05:58:02.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T05:58:02.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T05:58:02.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T05:58:02.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T05:58:03.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T05:58:03.154 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T05:58:03.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T05:58:03.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T05:58:03.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T05:58:03.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T05:58:03.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T05:58:03.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T05:58:03.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T05:58:03.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T05:58:03.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:58:03.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:58:03.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:58:03.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:58:03.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:58:03.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:58:03.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:58:03.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:58:04.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:58:04.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:58:04.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:58:04.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:58:04.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:58:04.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:58:04.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:58:04.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:58:04.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:58:04.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:58:04.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:58:04.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:58:04.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:58:04.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:58:04.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:58:04.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:04.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:04.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:58:04.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:58:04.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T05:58:04.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T05:58:04.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T05:58:04.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T05:58:04.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T05:58:04.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T05:58:04.137 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T05:58:04.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T05:58:04.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T05:58:04.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T05:58:04.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T05:58:04.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T05:58:04.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:58:04.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T05:58:04.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T05:58:04.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T05:58:04.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T05:58:04.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T05:58:04.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T05:58:04.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T05:58:04.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:04.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:04.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T05:58:04.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T05:58:04.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T05:58:04.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T05:58:04.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T05:58:04.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:04.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:04.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T05:58:04.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-29T05:58:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-29T05:58:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T05:58:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T05:58:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T05:58:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T05:58:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T05:58:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T05:58:04.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T05:58:04.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:58:04.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:58:04.554 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:58:04.554 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:04.555 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:04.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:58:04.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T05:58:04.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T05:58:04.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:58:04.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T05:58:04.873 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T05:58:04.874 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T05:58:04.874 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:04.874 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:04.874 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T05:58:04.875 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T05:58:04.875 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T05:58:04.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-29T05:58:04.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T05:58:04.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T05:58:04.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T05:58:04.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T05:58:04.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T05:58:04.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T05:58:05.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T05:58:05.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:58:05.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:58:05.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:58:05.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:05.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:05.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:58:05.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:58:05.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T05:58:05.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T05:58:05.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:05.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T05:58:05.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T05:58:05.875 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:58:05.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72bc6074-4cca-464b-8ec4-6c2e550f54d9 2024-03-29T05:58:05.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 72bc6074-4cca-464b-8ec4-6c2e550f54d9' 2024-03-29T05:58:05.875 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 72bc6074-4cca-464b-8ec4-6c2e550f54d9 2024-03-29T05:58:05.876 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:58:05.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtWAZmnsg/NRAAdGRYTs2SObb0m7L3h21HVA== 2024-03-29T05:58:05.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtWAZmnsg/NRAAdGRYTs2SObb0m7L3h21HVA=="}' 2024-03-29T05:58:05.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72bc6074-4cca-464b-8ec4-6c2e550f54d9 -i td/osd-scrub-repair/0/new.json 2024-03-29T05:58:06.126 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:58:06.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T05:58:06.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtWAZmnsg/NRAAdGRYTs2SObb0m7L3h21HVA== --osd-uuid 72bc6074-4cca-464b-8ec4-6c2e550f54d9 2024-03-29T05:58:06.172 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:06.172+0000 7fccf41137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:06.176 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:06.176+0000 7fccf41137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:06.177 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:06.176+0000 7fccf41137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:06.178 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:06.176+0000 7fccf41137c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:58:06.178 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:06.176+0000 7fccf41137c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:58:06.178 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:06.176+0000 7fccf41137c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T05:58:06.277 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:06.276+0000 7fccf41137c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T05:58:12.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T05:58:12.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:58:12.969 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T05:58:12.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T05:58:12.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:58:13.342 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T05:58:13.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T05:58:13.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:58:13.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:58:13.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:58:13.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:58:13.374 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:13.372+0000 7f49085e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:13.376 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:13.376+0000 7f49085e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:13.377 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:13.376+0000 7f49085e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:13.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T05:58:13.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:58:13.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T05:58:13.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:58:13.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:58:13.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:13.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:58:13.690 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:58:13.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:13.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:13.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:13.912+0000 7f49085e07c0 -1 Falling back to public interface 2024-03-29T05:58:14.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:15.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:15.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:15.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:58:15.025 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:58:15.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:15.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:15.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:16.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:16.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:16.347 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:58:16.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:58:16.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:16.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:16.586 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:16.584+0000 7f49085e07c0 -1 osd.0 0 log_to_monitors true 2024-03-29T05:58:16.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:17.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:17.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:17.707 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:58:17.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:58:17.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:17.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:18.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:19.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:19.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:19.027 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:58:19.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:58:19.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:19.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:19.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:20.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:20.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:20.350 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:58:20.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:58:20.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:20.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:20.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:21.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:21.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:21.670 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:58:21.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:58:21.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:21.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:21.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:22.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:22.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:22.989 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:58:22.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:58:22.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:22.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:23.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:24.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:24.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:24.295 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:58:24.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:58:24.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:24.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:24.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:25.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:25.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:25.669 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:58:25.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:58:25.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:25.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:25.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:26.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:26.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:26.987 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:58:26.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:58:26.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:26.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:27.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:28.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:28.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:28.317 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:58:28.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:58:28.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:28.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:28.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:29.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:29.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:29.643 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:58:29.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:58:29.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:29.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:29.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:30.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:30.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:30.969 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:58:30.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:58:30.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:30.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:31.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:32.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:32.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:32.724 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:58:32.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:58:32.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:32.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:33.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:34.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:34.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:34.466 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:58:34.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:58:34.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:34.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:35.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:36.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:36.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:36.407 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:58:36.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:58:36.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:36.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:37.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:38.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:38.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:38.295 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:58:38.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:58:38.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:38.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:38.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:40.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:40.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:40.001 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:58:40.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:58:40.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:40.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:40.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:41.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:41.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:41.641 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:58:41.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:58:41.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:41.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:42.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:43.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:43.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:43.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:58:43.028 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:58:43.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:43.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T05:58:43.344 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3799296503,v1:127.0.0.1:6803/3799296503] [v2:127.0.0.1:6804/3799296503,v1:127.0.0.1:6805/3799296503] exists,up 72bc6074-4cca-464b-8ec4-6c2e550f54d9 2024-03-29T05:58:43.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:58:43.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:58:43.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:58:43.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:58:43.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:58:43.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T05:58:43.356 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T05:58:43.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 78ac5e0c-e073-4220-872e-936bacad3c3d' 2024-03-29T05:58:43.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:58:43.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTWAZmbZotFhAAL348yLBlNkHwQAUkAG9BvQ== 2024-03-29T05:58:43.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTWAZmbZotFhAAL348yLBlNkHwQAUkAG9BvQ=="}' 2024-03-29T05:58:43.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78ac5e0c-e073-4220-872e-936bacad3c3d -i td/osd-scrub-repair/1/new.json 2024-03-29T05:58:43.940 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:58:43.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T05:58:43.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTWAZmbZotFhAAL348yLBlNkHwQAUkAG9BvQ== --osd-uuid 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T05:58:43.995 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:43.995+0000 7f12804d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:44.042 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:44.039+0000 7f12804d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:44.044 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:44.043+0000 7f12804d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:44.044 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:44.043+0000 7f12804d17c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:58:44.045 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:44.043+0000 7f12804d17c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:58:44.046 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:44.043+0000 7f12804d17c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T05:58:44.111 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:44.107+0000 7f12804d17c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T05:58:51.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T05:58:51.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:58:51.125 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T05:58:51.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T05:58:51.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:58:51.517 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T05:58:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T05:58:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:58:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:58:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:58:51.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:58:51.550 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:51.547+0000 7fe6e95857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:51.606 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:51.603+0000 7fe6e95857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:51.607 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:51.607+0000 7fe6e95857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:58:51.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T05:58:51.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:58:51.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T05:58:51.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:58:51.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:58:51.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:51.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:58:51.830 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:58:51.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:51.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:58:52.157 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:52.155+0000 7fe6e95857c0 -1 Falling back to public interface 2024-03-29T05:58:52.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:53.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:53.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:53.165 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:58:53.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:58:53.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:53.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:58:53.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:54.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:54.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:54.474 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:58:54.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:58:54.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:54.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:58:54.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:54.867 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:58:54.867+0000 7fe6e95857c0 -1 osd.1 0 log_to_monitors true 2024-03-29T05:58:55.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:55.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:55.790 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:58:55.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:58:55.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:55.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:58:56.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:57.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:57.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:57.147 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:58:57.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:58:57.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:57.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:58:57.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:58.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:58.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:58.532 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:58:58.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:58:58.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:58.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:58:58.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:58:59.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:58:59.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:58:59.992 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:58:59.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:58:59.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:58:59.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:00.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:01.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:01.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:01.307 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:59:01.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:59:01.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:01.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:01.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:02.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:02.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:02.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:59:02.721 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:59:02.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:02.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:03.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:04.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:04.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:04.256 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:59:04.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:59:04.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:04.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:04.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:05.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:05.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:05.627 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:59:05.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:59:05.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:05.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:06.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:07.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:07.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:07.086 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:59:07.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:59:07.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:07.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:07.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:08.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:08.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:08.441 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:59:08.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:59:08.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:08.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:08.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:09.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:09.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:09.763 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:59:09.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:59:09.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:09.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:10.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:11.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:11.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:11.346 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:59:11.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:59:11.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:11.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:12.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:12.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:12.913 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:59:12.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:59:12.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:12.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:13.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:14.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:14.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:14.467 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:59:14.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:59:14.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:14.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:14.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:15.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:15.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:15.930 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:59:15.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:59:15.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:15.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:16.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:17.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:17.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:17.469 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:59:17.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:59:17.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:17.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:17.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:18.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:18.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:18.976 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:59:18.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:59:18.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:18.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:19.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:20.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:20.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:20.522 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:59:20.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:59:20.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:20.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:20.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:21.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:21.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:21.921 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T05:59:21.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T05:59:21.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:21.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T05:59:22.247 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3752613541,v1:127.0.0.1:6811/3752613541] [v2:127.0.0.1:6812/3752613541,v1:127.0.0.1:6813/3752613541] exists,up 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T05:59:22.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T05:59:22.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T05:59:22.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T05:59:22.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T05:59:22.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T05:59:22.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T05:59:22.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T05:59:22.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T05:59:22.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T05:59:22.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T05:59:22.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T05:59:22.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6d99474b-f87a-4271-adfa-b17c084cfa98 2024-03-29T05:59:22.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6d99474b-f87a-4271-adfa-b17c084cfa98' 2024-03-29T05:59:22.254 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 6d99474b-f87a-4271-adfa-b17c084cfa98 2024-03-29T05:59:22.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T05:59:22.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6WAZm2Y+WEBAAK1QYzadC2Gth4500aL5DAA== 2024-03-29T05:59:22.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6WAZm2Y+WEBAAK1QYzadC2Gth4500aL5DAA=="}' 2024-03-29T05:59:22.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6d99474b-f87a-4271-adfa-b17c084cfa98 -i td/osd-scrub-repair/2/new.json 2024-03-29T05:59:22.687 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:59:22.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T05:59:22.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6WAZm2Y+WEBAAK1QYzadC2Gth4500aL5DAA== --osd-uuid 6d99474b-f87a-4271-adfa-b17c084cfa98 2024-03-29T05:59:22.747 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:22.746+0000 7f57763137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:59:22.814 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:22.814+0000 7f57763137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:59:22.816 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:22.818+0000 7f57763137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:59:22.817 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:22.818+0000 7f57763137c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:59:22.817 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:22.818+0000 7f57763137c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:59:22.817 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:22.818+0000 7f57763137c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T05:59:22.897 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:22.898+0000 7f57763137c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T05:59:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T05:59:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T05:59:29.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T05:59:29.775 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T05:59:29.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T05:59:30.167 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T05:59:30.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T05:59:30.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T05:59:30.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T05:59:30.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T05:59:30.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T05:59:30.199 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:30.198+0000 7ff215e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:59:30.201 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:30.202+0000 7ff215e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:59:30.202 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:30.202+0000 7ff215e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T05:59:30.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T05:59:30.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T05:59:30.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T05:59:30.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T05:59:30.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T05:59:30.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:30.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T05:59:30.503 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T05:59:30.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:30.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:30.738 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:30.738+0000 7ff215e5f7c0 -1 Falling back to public interface 2024-03-29T05:59:30.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:31.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:31.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:31.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T05:59:31.810 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T05:59:31.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:31.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:32.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:33.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:33.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:33.133 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T05:59:33.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T05:59:33.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:33.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:33.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:33.648 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T05:59:33.650+0000 7ff215e5f7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T05:59:34.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:34.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:34.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T05:59:34.456 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T05:59:34.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:34.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:34.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:35.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:35.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:35.948 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T05:59:35.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T05:59:35.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:35.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:36.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:37.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:37.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:37.265 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T05:59:37.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T05:59:37.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:37.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:37.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:38.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:38.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:38.624 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T05:59:38.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T05:59:38.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:38.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:38.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:39.932 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T05:59:39.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T05:59:39.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:39.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:40.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:41.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:41.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:41.237 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T05:59:41.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T05:59:41.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:41.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:41.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:42.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:42.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:42.595 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T05:59:42.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T05:59:42.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:42.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:43.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:44.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:44.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:44.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T05:59:44.050 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T05:59:44.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:44.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:44.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:45.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:45.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:45.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T05:59:45.420 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T05:59:45.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:45.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:45.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:46.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:46.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:46.880 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T05:59:46.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T05:59:46.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:46.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:47.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:48.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:48.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:48.292 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T05:59:48.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T05:59:48.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:48.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:49.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:50.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:50.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:50.205 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T05:59:50.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T05:59:50.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:50.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:50.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:51.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:51.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:51.984 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T05:59:51.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T05:59:51.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:51.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:52.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:53.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:53.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:53.572 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T05:59:53.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T05:59:53.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:53.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:54.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:55.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T05:59:55.009 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T05:59:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:55.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:55.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:56.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:56.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:56.504 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T05:59:56.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T05:59:56.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:56.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:56.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:57.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:57.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:57.877 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T05:59:57.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T05:59:57.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:57.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:58.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T05:59:59.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T05:59:59.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T05:59:59.506 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T05:59:59.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T05:59:59.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T05:59:59.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T05:59:59.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:00.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:00.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:00.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:00:00.846 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:00:00.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:00.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:00:01.172 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4231573471,v1:127.0.0.1:6819/4231573471] [v2:127.0.0.1:6820/4231573471,v1:127.0.0.1:6821/4231573471] exists,up 6d99474b-f87a-4271-adfa-b17c084cfa98 2024-03-29T06:00:01.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:00:01.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:00:01.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:00:01.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T06:00:01.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:00:01.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:00:01.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:00:01.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:00:01.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:00:01.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:00:01.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:00:01.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:00:01.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:00:01.177 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:00:01.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:00:01.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e0b6f1e6-dd56-4e69-980c-e61911bf8222' 2024-03-29T06:00:01.178 INFO:tasks.workunit.client.0.smithi123.stdout:add osd3 e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:00:01.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:00:01.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhWAZm5czyCxAAU/1bLZyBa+6WH4E6WhkB/g== 2024-03-29T06:00:01.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhWAZm5czyCxAAU/1bLZyBa+6WH4E6WhkB/g=="}' 2024-03-29T06:00:01.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e0b6f1e6-dd56-4e69-980c-e61911bf8222 -i td/osd-scrub-repair/3/new.json 2024-03-29T06:00:01.568 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:00:01.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-29T06:00:01.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhWAZm5czyCxAAU/1bLZyBa+6WH4E6WhkB/g== --osd-uuid e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:00:01.621 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:01.621+0000 7fb8488e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:00:01.678 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:01.677+0000 7fb8488e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:00:01.679 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:01.677+0000 7fb8488e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:00:01.680 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:01.681+0000 7fb8488e87c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:00:01.680 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:01.681+0000 7fb8488e87c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:00:01.680 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:01.681+0000 7fb8488e87c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:00:01.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:01.761+0000 7fb8488e87c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-29T06:00:08.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-29T06:00:08.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:00:08.726 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd3 key to auth repository 2024-03-29T06:00:08.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-29T06:00:08.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:00:09.110 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.3 2024-03-29T06:00:09.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-29T06:00:09.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:00:09.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:00:09.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:00:09.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:00:09.138 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:09.136+0000 7f8b13b867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:00:09.141 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:09.140+0000 7f8b13b867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:00:09.142 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:09.140+0000 7f8b13b867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:00:09.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-29T06:00:09.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:00:09.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:00:09.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:00:09.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:00:09.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:09.429 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:00:09.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:00:09.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:09.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:09.706 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:09.704+0000 7f8b13b867c0 -1 Falling back to public interface 2024-03-29T06:00:09.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:10.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:10.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:10.768 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:00:10.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:00:10.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:10.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:11.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:12.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:12.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:12.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:00:12.087 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:00:12.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:12.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:12.348 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:00:12.348+0000 7f8b13b867c0 -1 osd.3 0 log_to_monitors true 2024-03-29T06:00:12.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:13.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:13.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:13.402 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:00:13.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:00:13.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:13.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:13.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:14.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:14.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:14.722 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:00:14.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:00:14.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:14.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:15.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:16.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:16.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:16.045 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:00:16.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:00:16.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:16.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:16.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:17.354 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:00:17.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:00:17.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:17.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:17.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:18.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:18.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:18.781 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:00:18.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:00:18.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:18.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:19.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:20.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:20.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:20.098 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:00:20.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:00:20.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:20.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:20.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:21.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:21.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:21.641 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:00:21.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:00:21.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:21.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:21.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:22.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:22.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:22.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:00:22.961 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:00:22.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:22.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:23.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:24.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:24.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:24.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:00:24.300 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:00:24.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:24.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:24.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:25.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:25.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:00:25.780 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:00:25.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:25.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:26.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:27.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:27.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:27.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:00:27.085 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:00:27.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:27.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:27.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:28.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:28.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:28.796 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:00:28.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:00:28.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:28.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:29.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:30.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:30.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:30.824 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:00:30.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:00:30.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:30.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:31.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:32.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:32.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:32.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:00:32.300 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:00:32.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:32.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:32.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:33.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:33.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:33.860 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:00:33.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:00:33.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:33.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:34.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:35.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:35.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:35.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:00:35.572 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:00:35.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:35.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:35.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:36.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:36.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:36.879 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:00:36.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:00:36.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:36.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:37.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:00:38.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:00:38.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:00:38.350 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:00:38.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:00:38.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:00:38.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:00:38.745 INFO:tasks.workunit.client.0.smithi123.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3352966104,v1:127.0.0.1:6827/3352966104] [v2:127.0.0.1:6828/3352966104,v1:127.0.0.1:6829/3352966104] exists,up e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:00:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:00:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:00:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:00:38.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-29T06:00:38.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:00:39.047 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:00:39.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:00:39.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:00:39.503 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:00:39.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:00:40.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:00:43.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-29T06:00:43.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:00:43.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:00:43.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:00:43.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:00:43.147 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:00:43.147 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:00:43.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:00:43.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:00:43.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:00:43.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:00:43.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:00:43.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:00:43.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:00:43.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:00:43.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:00:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:00:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:00:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:00:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:00:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:00:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:43.589 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:00:43.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-29T06:00:43.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-29T06:00:43.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-29T06:00:43.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:43.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:00:43.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-29T06:00:43.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-29T06:00:43.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672984' 2024-03-29T06:00:43.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:43.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:00:44.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-29T06:00:44.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-29T06:00:44.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672984 2-64424509456' 2024-03-29T06:00:44.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:44.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:00:44.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-03-29T06:00:44.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-03-29T06:00:44.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672984 2-64424509456 3-85899345928' 2024-03-29T06:00:44.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:44.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-29T06:00:44.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:44.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:00:44.184 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-29T06:00:44.184 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:44.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-29T06:00:44.185 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836507 2024-03-29T06:00:44.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-29T06:00:44.185 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:00:44.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836507 2024-03-29T06:00:44.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:00:45.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:00:45.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:00:45.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-03-29T06:00:45.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:45.784 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-29T06:00:45.784 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:45.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:00:45.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-29T06:00:45.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:45.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-29T06:00:45.787 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672984 2024-03-29T06:00:45.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-29T06:00:45.788 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:00:46.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-03-29T06:00:46.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:46.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-29T06:00:46.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:46.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:00:46.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-29T06:00:46.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-29T06:00:46.134 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509456 2024-03-29T06:00:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-29T06:00:46.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:00:46.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-29T06:00:46.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:46.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345928 2024-03-29T06:00:46.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:46.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:00:46.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345928 2024-03-29T06:00:46.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:46.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-03-29T06:00:46.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-03-29T06:00:46.463 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345928 2024-03-29T06:00:46.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:00:46.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345928 2024-03-29T06:00:46.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:00:46.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:00:46.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:00:47.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:00:47.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:00:47.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:00:47.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:00:47.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:00:47.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:00:47.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:00:47.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:00:47.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:00:47.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:00:47.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:00:47.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:00:47.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:00:47.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:00:47.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:00:47.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-03-29T06:00:47.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T06:00:47.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T06:00:47.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-29T06:00:47.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T06:00:47.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-29T06:00:48.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T06:00:48.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T06:00:49.287 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T06:00:49.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:00:50.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-29T06:00:50.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T06:00:50.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:00:50.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:00:50.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:00:50.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:00:50.313 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:00:50.313 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:00:50.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:00:50.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:00:50.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:00:50.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:00:50.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:00:50.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:00:50.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:00:50.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:00:50.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:00:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:00:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:00:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:00:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:00:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:00:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:50.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:00:50.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-29T06:00:50.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-29T06:00:50.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-29T06:00:50.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:50.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:00:51.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-29T06:00:51.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-29T06:00:51.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672986' 2024-03-29T06:00:51.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:51.230 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:00:51.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-29T06:00:51.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-29T06:00:51.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672986 2-64424509459' 2024-03-29T06:00:51.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:00:51.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:00:51.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345930 2024-03-29T06:00:51.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345930 2024-03-29T06:00:51.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672986 2-64424509459 3-85899345930' 2024-03-29T06:00:51.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:51.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-29T06:00:51.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:51.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:00:51.518 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-29T06:00:51.518 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:51.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-29T06:00:51.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-29T06:00:51.521 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836509 2024-03-29T06:00:51.521 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:00:51.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-29T06:00:51.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:00:52.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:00:52.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:00:53.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-29T06:00:53.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:53.173 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:53.173 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-29T06:00:53.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:00:53.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-29T06:00:53.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:53.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-29T06:00:53.176 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672986 2024-03-29T06:00:53.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-29T06:00:53.177 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:00:53.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672986 2024-03-29T06:00:53.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:53.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-29T06:00:53.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:53.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:00:53.515 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-29T06:00:53.515 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:53.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-29T06:00:53.517 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509459 2024-03-29T06:00:53.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-29T06:00:53.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:00:53.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-29T06:00:53.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:00:53.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345930 2024-03-29T06:00:53.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:00:53.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:00:53.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345930 2024-03-29T06:00:53.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:00:53.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345930 2024-03-29T06:00:53.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-03-29T06:00:53.847 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345930 2024-03-29T06:00:53.848 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:00:54.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345930 -lt 85899345930 2024-03-29T06:00:54.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:00:54.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:00:54.172 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:00:54.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:00:54.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:00:54.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:00:54.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:00:54.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:00:54.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:00:54.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:00:54.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:00:54.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:00:54.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:00:54.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:00:54.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-29T06:00:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:00:55.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:00:55.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-29T06:00:55.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:00:55.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:00:55.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:00:55.787 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:00:55.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:00:56.206 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:00:56.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:00:56.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:00:56.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T06:00:56.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-29T06:00:56.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T06:00:56.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T06:00:56.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:00:56.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:00:56.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-29T06:00:56.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-29T06:00:56.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T06:00:56.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-29T06:00:56.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-29T06:00:56.988 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:00:56.988 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:00:57.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-29T06:00:57.301 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:00:57.302 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:00:57.302 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:00:57.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-29T06:00:57.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-29T06:00:57.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-29T06:00:57.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-29T06:00:57.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:00:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:00:57.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:00:57.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:00:57.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:00:57.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:00:57.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:00:57.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:00:59.539 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:01:00.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:01:00.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:01:00.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:01:00.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:01:00.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:01:00.088 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:01:00.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:01:00.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:01:00.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:01:00.090 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:01:00.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:01:00.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:01:00.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:01:00.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:01:00.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:01:00.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:01:00.123 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:00.119+0000 7f006a8667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:00.125 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:00.123+0000 7f006a8667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:00.127 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:00.123+0000 7f006a8667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:00.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:01:00.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:01:00.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:01:00.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:01:00.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:01:00.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:00.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:01:00.413 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:01:00.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:00.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:00.663 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:00.659+0000 7f006a8667c0 -1 Falling back to public interface 2024-03-29T06:01:00.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:01.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:01.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:01.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:01:01.751 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:01.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:01.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:02.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:03.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:03.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:03.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:01:03.064 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:03.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:03.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:03.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:03.536 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:03.511+0000 7f006a8667c0 -1 osd.3 35 log_to_monitors true 2024-03-29T06:01:04.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:04.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:04.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:01:04.380 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:01:04.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:04.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/3642637652,v1:127.0.0.1:6827/3642637652] [v2:127.0.0.1:6828/3642637652,v1:127.0.0.1:6829/3642637652] exists,up e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:01:04.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:01:04.737 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:01:04.737 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:01:04.737 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:01:04.737 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:01:04.737 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:01:04.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:01:04.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:01:04.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:01:04.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:01:04.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:01:04.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:01:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:01:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:05.244 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:01:05.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:01:05.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:05.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:01:05.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-03-29T06:01:05.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-03-29T06:01:05.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-03-29T06:01:05.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:05.390 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:01:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-03-29T06:01:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-03-29T06:01:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672990' 2024-03-29T06:01:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:05.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:01:05.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-29T06:01:05.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-29T06:01:05.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672990 2-64424509463' 2024-03-29T06:01:05.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:05.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:01:06.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-29T06:01:06.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-29T06:01:06.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672990 2-64424509463 3-167503724546' 2024-03-29T06:01:06.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:06.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-03-29T06:01:06.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:06.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:01:06.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-03-29T06:01:06.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:06.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-03-29T06:01:06.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-03-29T06:01:06.102 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836513 2024-03-29T06:01:06.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:06.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-03-29T06:01:06.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:01:07.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:01:07.393 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:07.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-03-29T06:01:07.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:07.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-03-29T06:01:07.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:07.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:01:07.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-03-29T06:01:07.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-03-29T06:01:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-03-29T06:01:07.719 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672990 2024-03-29T06:01:07.719 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:01:08.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672990 2024-03-29T06:01:08.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:08.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-29T06:01:08.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:08.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:01:08.037 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-29T06:01:08.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:08.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-29T06:01:08.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-29T06:01:08.039 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509463 2024-03-29T06:01:08.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:01:08.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-29T06:01:08.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:08.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:08.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-167503724546 2024-03-29T06:01:08.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:01:08.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-167503724546 2024-03-29T06:01:08.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:08.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-29T06:01:08.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 167503724546' 2024-03-29T06:01:08.385 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 167503724546 2024-03-29T06:01:08.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:01:08.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-29T06:01:08.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:01:08.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:01:08.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:01:09.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:01:09.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:01:09.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:01:09.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:01:09.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:01:09.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:01:09.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:01:09.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:01:09.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:01:09.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:01:09.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:01:09.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:01:09.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:01:09.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:01:09.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:01:09.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:01:09.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:01:09.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:01:09.830 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:01:09.830 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:01:10.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:01:10.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:01:10.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:01:10.172 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:01:10.172 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:10.173 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:10.173 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:10.173 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:10.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:00:49.092065+0000 2024-03-29T06:01:10.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:01:10.774 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:01:10.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:00:49.092065+0000 2024-03-29T06:01:10.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:01:10.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:00:49.092065+0000 2024-03-29T06:01:10.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:01:10.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:01:10.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:10.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:10.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:10.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:10.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:10.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:11.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:00:49.092065+0000 '>' 2024-03-29T06:00:49.092065+0000 2024-03-29T06:01:11.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:12.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:12.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:12.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:12.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:12.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:12.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:12.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:12.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:00:49.092065+0000 '>' 2024-03-29T06:00:49.092065+0000 2024-03-29T06:01:12.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:13.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:13.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:13.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:13.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:13.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:13.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:13.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:13.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:00:49.092065+0000 '>' 2024-03-29T06:00:49.092065+0000 2024-03-29T06:01:13.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:14.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:14.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:14.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:14.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:14.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:15.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:11.575475+0000 '>' 2024-03-29T06:00:49.092065+0000 2024-03-29T06:01:15.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:01:15.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:01:15.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:01:15.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:01:16.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:01:16.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:01:16.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:01:16.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:01:16.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:01:16.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:01:16.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:01:16.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:01:17.687 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:01:17.688 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:01:17.688 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:01:17.688 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:01:18.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:01:18.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:01:18.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:01:18.225 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:01:18.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:01:18.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:01:18.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:01:18.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:01:18.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:01:18.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:01:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:18.254+0000 7f1f3c09f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:18.262 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:18.258+0000 7f1f3c09f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:18.264 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:18.258+0000 7f1f3c09f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:01:18.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:18.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:18.814 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:18.810+0000 7f1f3c09f7c0 -1 Falling back to public interface 2024-03-29T06:01:18.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:19.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:19.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:19.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:01:19.924 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:19.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:19.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:20.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:21.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:21.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:21.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:01:21.244 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:21.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:21.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:21.428 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:21.430+0000 7f1f3c09f7c0 -1 osd.3 42 log_to_monitors true 2024-03-29T06:01:21.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:22.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:22.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:22.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:01:22.578 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:01:22.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:22.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [39,42) [v2:127.0.0.1:6826/3935726328,v1:127.0.0.1:6827/3935726328] [v2:127.0.0.1:6828/3935726328,v1:127.0.0.1:6829/3935726328] exists,up e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:01:23.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:01:23.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:01:23.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:01:23.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:01:23.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:01:23.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:01:23.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:01:23.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:01:23.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:01:23.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:01:23.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:01:23.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:01:23.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:01:23.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:01:23.568 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:23.568 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:23.568 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:01:23.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:01:23.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:23.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:01:23.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-03-29T06:01:23.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-03-29T06:01:23.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-03-29T06:01:23.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:23.740 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:01:23.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672995 2024-03-29T06:01:23.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672995 2024-03-29T06:01:23.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672995' 2024-03-29T06:01:23.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:23.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:01:24.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-29T06:01:24.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-29T06:01:24.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672995 2-64424509467' 2024-03-29T06:01:24.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:24.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:01:24.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-03-29T06:01:24.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-03-29T06:01:24.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672995 2-64424509467 3-193273528322' 2024-03-29T06:01:24.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:24.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-03-29T06:01:24.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:24.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:01:24.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-03-29T06:01:24.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:24.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-03-29T06:01:24.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-03-29T06:01:24.170 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836518 2024-03-29T06:01:24.170 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:24.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836518 2024-03-29T06:01:24.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:01:25.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:01:25.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836518 2024-03-29T06:01:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:25.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672995 2024-03-29T06:01:25.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:25.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:01:25.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672995 2024-03-29T06:01:25.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:25.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672995 2024-03-29T06:01:25.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-03-29T06:01:25.787 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672995 2024-03-29T06:01:25.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:01:26.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672995 -lt 42949672995 2024-03-29T06:01:26.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:26.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-29T06:01:26.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:26.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:01:26.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-29T06:01:26.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:26.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-29T06:01:26.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-29T06:01:26.164 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509467 2024-03-29T06:01:26.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:01:26.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-29T06:01:26.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:26.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-193273528322 2024-03-29T06:01:26.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:26.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:01:26.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-193273528322 2024-03-29T06:01:26.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:26.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-03-29T06:01:26.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 193273528322' 2024-03-29T06:01:26.492 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 193273528322 2024-03-29T06:01:26.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:01:26.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-03-29T06:01:26.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:01:26.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:01:26.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:01:27.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:01:27.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:01:27.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:01:27.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:01:27.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:01:27.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:01:27.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:01:27.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:01:27.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:01:27.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:01:27.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:01:27.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:01:27.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:01:27.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:01:27.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:01:27.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:01:28.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:01:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-29T06:01:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:01:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:01:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-29T06:01:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:01:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:01:28.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:01:29.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:01:29.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:01:29.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:01:29.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:01:29.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:01:29.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:01:29.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:01:29.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T06:01:30.941 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:01:31.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:01:31.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:01:31.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:01:31.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:01:31.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:01:31.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:01:31.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:01:31.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:01:31.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:01:31.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:01:31.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:01:31.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:01:31.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:01:31.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:01:31.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:01:31.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:01:31.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:01:31.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:01:31.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:01:31.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:01:31.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:01:31.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:01:31.513 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:31.514+0000 7fc24d2f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:31.516 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:31.514+0000 7fc24d2f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:31.517 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:31.518+0000 7fc24d2f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:31.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:01:31.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:32.056 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:32.054+0000 7fc24d2f57c0 -1 Falling back to public interface 2024-03-29T06:01:32.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:33.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:33.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:33.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:01:33.149 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:33.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:33.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:01:33.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:34.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:34.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:34.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:01:34.469 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:34.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:34.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:01:34.600 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:34.602+0000 7fc24d2f57c0 -1 osd.1 47 log_to_monitors true 2024-03-29T06:01:34.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:35.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:35.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:35.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:01:35.805 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:01:35.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:35.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 48 last_clean_interval [10,47) [v2:127.0.0.1:6810/950886459,v1:127.0.0.1:6811/950886459] [v2:127.0.0.1:6812/950886459,v1:127.0.0.1:6813/950886459] exists,up 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:01:36.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:01:36.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:01:36.468 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:01:36.468 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:01:36.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:01:36.469 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:01:36.469 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:01:36.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:01:36.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:01:36.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:01:36.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:01:36.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:01:36.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:01:36.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:01:36.969 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:36.969 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:36.969 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:01:36.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:01:36.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:36.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:01:37.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836521 2024-03-29T06:01:37.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836521 2024-03-29T06:01:37.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521' 2024-03-29T06:01:37.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:37.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:01:37.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299394 2024-03-29T06:01:37.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299394 2024-03-29T06:01:37.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-223338299394' 2024-03-29T06:01:37.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:37.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:01:37.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-03-29T06:01:37.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-03-29T06:01:37.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-223338299394 2-64424509471' 2024-03-29T06:01:37.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:37.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:01:37.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528326 2024-03-29T06:01:37.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528326 2024-03-29T06:01:37.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-223338299394 2-64424509471 3-193273528326' 2024-03-29T06:01:37.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:37.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836521 2024-03-29T06:01:37.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:37.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:01:37.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836521 2024-03-29T06:01:37.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:37.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836521 2024-03-29T06:01:37.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-03-29T06:01:37.637 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836521 2024-03-29T06:01:37.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:37.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836521 2024-03-29T06:01:37.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:01:38.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:01:38.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:39.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836521 2024-03-29T06:01:39.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:39.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299394 2024-03-29T06:01:39.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:39.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:01:39.280 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299394 2024-03-29T06:01:39.280 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:39.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299394 2024-03-29T06:01:39.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299394' 2024-03-29T06:01:39.281 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 223338299394 2024-03-29T06:01:39.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:01:39.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299394 -lt 223338299394 2024-03-29T06:01:39.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:39.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-03-29T06:01:39.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:39.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:01:39.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-03-29T06:01:39.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:39.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-03-29T06:01:39.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-03-29T06:01:39.601 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509471 2024-03-29T06:01:39.602 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:01:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-03-29T06:01:39.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:39.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-193273528326 2024-03-29T06:01:39.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:39.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:01:39.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-193273528326 2024-03-29T06:01:39.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:39.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528326 2024-03-29T06:01:39.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 193273528326' 2024-03-29T06:01:39.939 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 193273528326 2024-03-29T06:01:39.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:01:40.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528326 -lt 193273528326 2024-03-29T06:01:40.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:01:40.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:01:40.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:01:40.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:01:40.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:01:40.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:01:40.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:01:40.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:01:40.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:01:40.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:01:40.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:01:41.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:01:41.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:01:41.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:01:41.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:01:41.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:01:41.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:01:41.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:01:41.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:01:41.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:01:41.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:01:41.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:01:41.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:01:41.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:01:41.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:01:41.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:01:41.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:01:41.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:41.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:41.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:41.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:42.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:42.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:01:42.343 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:01:42.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:42.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:42.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:42.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:11.575475+0000 '>' 2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:42.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:43.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:43.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:43.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:43.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:43.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:43.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:43.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:43.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:11.575475+0000 '>' 2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:43.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:44.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:44.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:44.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:44.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:44.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:44.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:44.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:45.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:11.575475+0000 '>' 2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:45.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:46.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:46.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:46.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:46.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:46.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:46.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:46.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:46.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:11.575475+0000 '>' 2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:46.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:47.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:47.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:47.578 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:47.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:47.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:47.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:47.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:47.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:11.575475+0000 '>' 2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:47.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:01:48.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:01:48.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:01:48.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:01:48.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:01:48.889 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:01:48.889 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:01:48.889 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:43.325360+0000 '>' 2024-03-29T06:01:11.575475+0000 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:01:49.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:01:49.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:01:49.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:01:51.586 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:01:51.586 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:01:51.586 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:01:51.586 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:01:51.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:01:51.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:01:51.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:01:51.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:01:51.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:01:51.886 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:01:51.887 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:01:51.887 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:01:51.887 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:01:51.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:51.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:01:51.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:01:51.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:01:51.889 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:01:51.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:01:51.889 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:01:51.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:01:51.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:01:51.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:01:51.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:01:51.929 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:51.929+0000 7ffaf78307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:51.931 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:51.929+0000 7ffaf78307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:51.933 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:51.933+0000 7ffaf78307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:01:52.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:52.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:52.483 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:52.481+0000 7ffaf78307c0 -1 Falling back to public interface 2024-03-29T06:01:52.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:53.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:53.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:53.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:01:53.575 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:53.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:53.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:53.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:54.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:54.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:54.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:01:54.900 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:54.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:54.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:54.995 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:01:54.993+0000 7ffaf78307c0 -1 osd.3 54 log_to_monitors true 2024-03-29T06:01:55.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:01:56.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:01:56.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:01:56.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:01:56.216 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:01:56.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:01:56.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 55 last_clean_interval [45,54) [v2:127.0.0.1:6826/2944103095,v1:127.0.0.1:6827/2944103095] [v2:127.0.0.1:6828/2944103095,v1:127.0.0.1:6829/2944103095] exists,up e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:01:56.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:01:56.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:01:56.659 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:01:56.659 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:01:56.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:01:56.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:01:56.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:01:56.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:01:56.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:01:56.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:01:56.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:01:56.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:01:56.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:01:57.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:01:57.108 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:01:57.108 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:01:57.108 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:01:57.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:01:57.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:57.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:01:57.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836526 2024-03-29T06:01:57.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836526 2024-03-29T06:01:57.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526' 2024-03-29T06:01:57.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:57.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:01:57.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299399 2024-03-29T06:01:57.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299399 2024-03-29T06:01:57.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-223338299399' 2024-03-29T06:01:57.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:57.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:01:57.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509476 2024-03-29T06:01:57.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509476 2024-03-29T06:01:57.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-223338299399 2-64424509476' 2024-03-29T06:01:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:01:57.538 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:01:57.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-03-29T06:01:57.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-03-29T06:01:57.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-223338299399 2-64424509476 3-253403070466' 2024-03-29T06:01:57.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:57.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836526 2024-03-29T06:01:57.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:57.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:01:57.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836526 2024-03-29T06:01:57.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:57.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836526 2024-03-29T06:01:57.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-03-29T06:01:57.680 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836526 2024-03-29T06:01:57.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836526 2024-03-29T06:01:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:01:58.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:01:58.986 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:01:59.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836526 2024-03-29T06:01:59.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:59.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299399 2024-03-29T06:01:59.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:59.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:01:59.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299399 2024-03-29T06:01:59.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:59.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299399 2024-03-29T06:01:59.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299399' 2024-03-29T06:01:59.303 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 223338299399 2024-03-29T06:01:59.303 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:01:59.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299399 -lt 223338299399 2024-03-29T06:01:59.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:59.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509476 2024-03-29T06:01:59.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:59.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:01:59.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509476 2024-03-29T06:01:59.637 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:59.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509476 2024-03-29T06:01:59.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-03-29T06:01:59.638 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509476 2024-03-29T06:01:59.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:01:59.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509476 -lt 64424509476 2024-03-29T06:01:59.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:01:59.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-03-29T06:01:59.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:01:59.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:01:59.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-03-29T06:01:59.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:01:59.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-03-29T06:01:59.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-03-29T06:01:59.948 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 253403070466 2024-03-29T06:01:59.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:02:00.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-03-29T06:02:00.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:02:00.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:00.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:00.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:00.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:02:00.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:00.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:00.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:01.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:02:01.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:02:01.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:02:01.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:02:01.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:02:01.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-29T06:02:01.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 158239"' 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 158239' 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 158240"' 2024-03-29T06:02:01.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 158240' 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 158239 158240' 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 158239 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/158242: /' 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:02:01.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/158244: /' 2024-03-29T06:02:04.757 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:02:04.758 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.759 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:02:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:02:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:02:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:02:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:02:04.760 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:02:04.762 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:02:04.762 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:02:04.762 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:02:04.762 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:02:04.762 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:02:04.762 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:02:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:02:04.764 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:02:04.764 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:02:04.764 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:02:04.764 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:02:04.764 INFO:tasks.workunit.client.0.smithi123.stderr:158242: start osd.1 2024-03-29T06:02:04.764 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:02:04.867 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:02:04.868 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:02:04.869 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:02:04.870 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:02:04.870 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:02:04.870 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:02:04.871 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: start osd.2 2024-03-29T06:02:04.872 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:02:09.471 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:02:09.471 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:02:09.471 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:02:09.471 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 2024-03-29T06:02:04.793+0000 7f083d9207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 2024-03-29T06:02:04.797+0000 7f083d9207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 2024-03-29T06:02:04.801+0000 7f083d9207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 0 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.472 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 2024-03-29T06:02:05.349+0000 7f083d9207c0 -1 Falling back to public interface 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 1 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 2 2024-03-29T06:02:09.473 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 2024-03-29T06:02:08.437+0000 7f083d9207c0 -1 osd.1 60 log_to_monitors true 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 3 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158242: osd.1 up in weight 1 up_from 66 up_thru 55 down_at 61 last_158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:02:09.474 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 2024-03-29T06:02:04.897+0000 7fcbcfae37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 2024-03-29T06:02:04.901+0000 7fcbcfae37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 2024-03-29T06:02:04.901+0000 7fcbcfae37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 0 2024-03-29T06:02:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 2024-03-29T06:02:05.449+0000 7fcbcfae37c0 -1 Falling back to public interface 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 1 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:02:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 2 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 2024-03-29T06:02:08.745+0000 7fcbcfae37c0 -1 osd.2 60 log_to_monitors true 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 3 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [52,60) [v2:127.0.0.1:6810/629776194,v1:127.0.0.1:6811/629776194] [v2:127.0.0.1:6812/629776194,v1:127.0.0.1:6813/629776194] exists,up 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:02:10.615 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 1 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 2 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 3' 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-283467841538' 2024-03-29T06:02:10.617 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tells.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 4 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: osd.2 up in weight 1 up_from 67 up_thru 48 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6818/2059846344,v1:127.0.0.1:6819/2059846344] [v2:127.0.0.1:6820/2059846344,v1:127.0.0.1:6821/2059846344] exists,up 6d99474b-f87a-4271-adfa-b17c084cfa98 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:02:11.646 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:02:11.647 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 1 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 2 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: 3' 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-03-29T06:02:11.648 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:12.813 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:02:12.813 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841539' 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841539 2-287762808835' 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070471 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070471 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-283467841539 2-287762808835 3-253403070471' 2024-03-29T06:02:12.814 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: waiting osd.0 seq 21474836531 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836531 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841539 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:12.815 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:02:12.816 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841539 2024-03-29T06:02:12.816 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:12.816 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-03-29T06:02:12.816 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-03-29T06:02:12.816 INFO:tasks.workunit.client.0.smithi123.stderr:158244: waiting osd.1 seq 283467841539 2024-03-29T06:02:12.816 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:02:13.292 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221 osd.2 flush_pg_stats 2024-03-29T06:02:13.292 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-29T06:02:13.292 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-283467841538 2-287762808834' 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-283467841538 2-287762808834 3-253403070470' 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-03-29T06:02:13.293 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: waiting osd.0 seq 21474836530 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836530 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:02:13.294 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-03-29T06:02:13.295 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:13.295 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-29T06:02:13.295 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-03-29T06:02:13.295 INFO:tasks.workunit.client.0.smithi123.stderr:158242: waiting osd.1 seq 283467841538 2024-03-29T06:02:13.295 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:02:13.295 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-03-29T06:02:13.295 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158242: 4: flush_pg_stats: test 283467841539 -lt 283467841539 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808835 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808835 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: waiting osd.2 seq 287762808835 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:02:14.240 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070471 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070471 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070471 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070471' 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: waiting osd.3 seq 253403070471 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070471 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:14.241 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:14.242 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:02:14.242 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:14.242 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:14.242 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:14.242 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:14.242 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:14.242 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:14.243 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:14.567 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clea//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808834 2024-03-29T06:02:14.567 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808834 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: waiting osd.2 seq 287762808834 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-03-29T06:02:14.568 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: waiting osd.3 seq 253403070470 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:02:14.569 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070470 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:14.570 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:14.571 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:14.571 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:14.571 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:14.571 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:02:14.571 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:14.634 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.clienn: cur_active_clean=5 2024-03-29T06:02:14.634 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:14.635 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:14.635 INFO:tasks.workunit.client.0.smithi123.stderr:158244: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:14.635 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:02:14.635 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:02:14.635 INFO:tasks.workunit.client.0.smithi123.stderr:158244: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:02:14.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:02:14.959 INFO:tasks.workunit.client.0.smithi123.stderr:t.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:14.959 INFO:tasks.workunit.client.0.smithi123.stderr:158242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:14.959 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:158242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 158240 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:02:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:02:14.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:02:14.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:02:14.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:02:14.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:02:14.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:02:15.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:02:15.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:02:15.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:02:15.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:02:15.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:15.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:15.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:15.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:15.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:15.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:02:15.845 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:15.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:15.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:15.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:16.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:43.325360+0000 '>' 2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:16.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:02:17.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:02:17.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:17.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:17.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:17.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:17.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:17.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:17.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:43.325360+0000 '>' 2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:17.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:02:18.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:02:18.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:18.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:18.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:18.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:18.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:18.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:18.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:43.325360+0000 '>' 2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:18.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:02:19.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:02:19.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:19.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:19.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:19.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:19.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:19.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:20.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:01:43.325360+0000 '>' 2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:20.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:02:21.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:02:21.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:21.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:21.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:21.136 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:21.136 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:21.136 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:02:16.043902+0000 '>' 2024-03-29T06:01:43.325360+0000 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 161093"' 2024-03-29T06:02:21.449 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 161093' 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/161095: /' 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 161096"' 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 161096' 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 161093 161096' 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 161093 2024-03-29T06:02:21.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:02:21.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/161099: /' 2024-03-29T06:02:24.362 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:02:24.363 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: _ 2024-03-29T06:02:24.364 INFO:tasks.workunit.client.0.smithi123.stderr:161095: hinfo_key 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: snapset 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:02:24.365 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:02:24.368 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:02:24.368 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:02:24.368 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:02:24.368 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:02:24.368 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:02:24.368 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:02:24.369 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:02:24.370 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:02:24.370 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:02:24.370 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:02:24.370 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:02:24.370 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:02:24.370 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:02:24.370 INFO:tasks.workunit.client.0.smithi123.stderr:161095: start osd.1 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:02:24.637 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:02:24.638 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: _ 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: hinfo_key 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: snapset 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:02:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:02:24.641 INFO:tasks.workunit.client.0.smithi123.stderr:161099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:02:24.642 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:02:24.643 INFO:tasks.workunit.client.0.smithi123.stderr:161099: start osd.2 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 2024-03-29T06:02:24.401+0000 7f00639eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:29.051 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 2024-03-29T06:02:24.405+0000 7f00639eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 2024-03-29T06:02:24.405+0000 7f00639eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 0 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 2024-03-29T06:02:24.941+0000 7f00639eb7c0 -1 Falling back to public interface 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:29.052 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 1 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 2 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 2024-03-29T06:02:27.488+0000 7f00639eb7c0 -1 osd.1 70 log_to_monitors true 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.054 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:02:29.054 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 3 2024-03-29T06:02:29.054 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.054 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:29.513 INFO:tasks.workunit.client.0.smithi123.stderr:161095: osd.1 up in weight 1 up_from 74 up_thru 74 down_aovery_ops 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 2024-03-29T06:02:24.669+0000 7f2225be47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 2024-03-29T06:02:24.673+0000 7f2225be47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 2024-03-29T06:02:24.677+0000 7f2225be47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:02:29.514 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 0 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 2024-03-29T06:02:25.217+0000 7f2225be47c0 -1 Falling back to public interface 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 1 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.515 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 2 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 2024-03-29T06:02:28.384+0000 7f2225be47c0 -1 osd.2 69 log_to_monitors true 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:02:29.516 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 3 2024-03-29T06:02:29.517 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:29.517 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:30.385 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ct 70 last_clean_interval [66,70) [v2:127.0.0.1:6810/4051522562,v1:127.0.0.1:6811/4051522562] [v2:127.0.0.1:6812/4051522562,v1:127.0.0.1:6813/4051522562] exists,up 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T06:02:30.385 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:02:30.386 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 1 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 2 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: 3' 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-03-29T06:02:30.387 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:30.388 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:02:30.388 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-29T06:02:30.388 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-29T06:02:30.388 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-317827579906' 2024-03-29T06:02:30.388 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_staeph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:02:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 4 2024-03-29T06:02:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: osd.2 up in weight 1 up_from 78 up_thru 48 down_at 70 last_clean_interval [67,69) [v2:127.0.0.1:6818/2063022173,v1:127.0.0.1:6819/2063022173] [v2:127.0.0.1:6820/2063022173,v1:127.0.0.1:6821/2063022173] exists,up 6d99474b-f87a-4271-adfa-b17c084cfa98 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:02:31.541 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 1 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 2 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: 3' 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-03-29T06:02:31.542 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:31.698 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalonts: ceph tell osd.2 flush_pg_stats 2024-03-29T06:02:31.698 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-03-29T06:02:31.698 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-03-29T06:02:31.698 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-317827579906 2-335007449090' 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-317827579906 2-335007449090 3-253403070476' 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: waiting osd.0 seq 21474836536 2024-03-29T06:02:31.699 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836536 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579906 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579906 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579906' 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: waiting osd.1 seq 317827579906 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:31.700 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449090 2024-03-29T06:02:31.701 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:31.701 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:02:31.701 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449090 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ue/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579907' 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449091 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449091 2024-03-29T06:02:33.628 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579907 2-335007449091' 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070477 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070477 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-317827579907 2-335007449091 3-253403070477' 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-03-29T06:02:33.629 INFO:tasks.workunit.client.0.smithi123.stderr:161099: waiting osd.0 seq 21474836537 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836537 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579907 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579907 2024-03-29T06:02:33.630 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:16109buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449090' 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: waiting osd.2 seq 335007449090 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 335007449090 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:02:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449091 -lt 335007449090 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: waiting osd.3 seq 253403070476 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070476 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:02:34.355 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:34.754 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:34.755 INFO:tasks.workunit.client.0.smithi123.stderr:161095: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:34.755 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:02:34.755 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:02:34.755 INFO:tasks.workunit.client.0.smithi123.stderr:161095: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:02:34.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:02:34.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:02:34.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 161096 2024-03-29T06:02:34.924 INFO:tasks.workunit.client.0.smithi123.stderr:9: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-03-29T06:02:34.924 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-03-29T06:02:34.924 INFO:tasks.workunit.client.0.smithi123.stderr:161099: waiting osd.1 seq 317827579907 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449091 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449091 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449091 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449091' 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: waiting osd.2 seq 335007449091 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:02:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449091 -lt 335007449091 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070477 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070477 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070477 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: waiting osd.3 seq 253403070477 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070477 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:02:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:34.927 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:34.927 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:02:34.927 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:34.927 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:34.927 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:34.927 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:35.608 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:35.608 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:35.608 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:161099: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:161099: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:02:35.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:02:35.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:02:35.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-29T06:02:35.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:02:35.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:02:35.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-29T06:02:35.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-29T06:02:35.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:02:35.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 163796"' 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 163796' 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 163797"' 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 163797' 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 163796 163797' 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:02:35.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:02:35.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 163796 2024-03-29T06:02:35.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/163799: /' 2024-03-29T06:02:35.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:35.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/163801: /' 2024-03-29T06:02:38.771 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:02:38.771 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:02:38.772 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:02:38.773 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:02:38.776 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:38.777 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: start osd.1 2024-03-29T06:02:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:02:40.159 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:02:40.159 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:02:40.159 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:02:40.159 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:02:40.159 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:02:40.159 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:02:40.159 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:02:40.160 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:02:40.161 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:02:40.164 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:02:40.164 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:02:40.164 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:02:40.164 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:02:40.164 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:02:40.164 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:02:40.164 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:02:40.165 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:02:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:02:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:02:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:02:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:02:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:02:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:163799: start osd.3 2024-03-29T06:02:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 2024-03-29T06:02:38.812+0000 7fc99b42f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 2024-03-29T06:02:38.816+0000 7fc99b42f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 2024-03-29T06:02:38.816+0000 7fc99b42f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:02:43.637 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 0 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 2024-03-29T06:02:39.356+0000 7fc99b42f7c0 -1 Falling back to public interface 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:02:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 1 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 2 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 2024-03-29T06:02:42.072+0000 7fc99b42f7c0 -1 osd.1 79 log_to_monitors true 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:02:43.639 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 3 2024-03-29T06:02:43.640 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:43.640 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:02:44.661 INFO:tasks.workunit.client.0.smithi123.stderr:163801: osd.1 up in weight 1 up_from 84 up_thru 78 down_at 80 last_clean_interval [74,79) [v2:127.0.0.1:6810/3028831367,v1:127.0.0.1:6811/3028831367] [v2:127.0.0.1:6812/3028831367,v1:127.0.0.1:6813/3028831367] exists,up 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T06:02:44.661 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:02:44.661 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:02:44.662 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 1 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 2 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: 3' 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-03-29T06:02:44.663 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-03-29T06:02:44.664 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:44.664 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:02:44.664 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-29T06:02:44.664 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-29T06:02:44.664 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-360777252866' 2024-03-29T06:02:44.664 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph 163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:02:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:02:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:02:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:02:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:02:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 2024-03-29T06:02:40.196+0000 7f7d730307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 2024-03-29T06:02:40.196+0000 7f7d730307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 2024-03-29T06:02:40.200+0000 7f7d730307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 0 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 2024-03-29T06:02:40.744+0000 7f7d730307c0 -1 Falling back to public interface 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:45.116 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 1 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 2 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 2024-03-29T06:02:44.168+0000 7f7d730307c0 -1 osd.3 79 log_to_monitors true 2024-03-29T06:02:45.117 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:45.118 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:45.118 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:02:45.118 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 3 2024-03-29T06:02:45.118 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:45.118 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:02:47.324 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:02:47.324 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:02:47.324 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:02:47.324 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 4 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: osd.3 up in weight 1 up_from 88 up_thru 88 down_at 80 last_clean_interval [59,79) [v2:127.0.0.1:6826/1485961659,v1:127.0.0.1:6827/1485961659] [v2:127.0.0.1:6828/1485961659,v1:127.0.0.1:6829/1485961659] exists,up e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:02:47.325 INFO:tasks.workunit.client.0.smithi123.stderr:163799: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 1 2024-03-29T06:02:47.326 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 2 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: 3' 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-03-29T06:02:47.327 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heltell osd.2 flush_pg_stats 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449095 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449095 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-360777252866 2-335007449095' 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: Error ENXIO: problem getting command descriptions from osd.3 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:02:47.344 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: waiting osd.0 seq 21474836541 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836541 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-360777252866 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:02:47.345 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-360777252866 2024-03-29T06:02:47.346 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:47.346 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-29T06:02:47.346 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 360777252866' 2024-03-29T06:02:47.346 INFO:tasks.workunit.client.0.smithi123.stderr:163801: waiting osd.1 seq 360777252866 2024-03-29T06:02:47.346 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:02:47.346 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-03-29T06:02:47.346 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:48.810 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449095 2024-03-29T06:02:48.810 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:48.810 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:02:48.810 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449095 2024-03-29T06:02:48.810 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449095 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449095' 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: waiting osd.2 seq 335007449095 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449095 -lt 335007449095 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:48.811 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:48.812 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:49.382 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalopers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:02:49.382 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252867 2024-03-29T06:02:49.382 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252867 2024-03-29T06:02:49.382 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-360777252867' 2024-03-29T06:02:49.382 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449096 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449096 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-360777252867 2-335007449096' 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-360777252867 2-335007449096 3-377957122050' 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-03-29T06:02:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: waiting osd.0 seq 21474836542 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836542 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836542 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-360777252867 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-360777252867 2024-03-29T06:02:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:49.471 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/une/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:49.471 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:49.471 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:49.471 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:49.471 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:49.471 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:02:49.471 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:49.472 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:49.472 INFO:tasks.workunit.client.0.smithi123.stderr:163801: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:49.472 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:02:49.472 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:02:49.472 INFO:tasks.workunit.client.0.smithi123.stderr:163801: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:02:49.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:02:50.658 INFO:tasks.workunit.client.0.smithi123.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252867 2024-03-29T06:02:50.658 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 360777252867' 2024-03-29T06:02:50.658 INFO:tasks.workunit.client.0.smithi123.stderr:163799: waiting osd.1 seq 360777252867 2024-03-29T06:02:50.658 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252868 -lt 360777252867 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449096 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449096 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449096 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449096' 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: waiting osd.2 seq 335007449096 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449096 -lt 335007449096 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-377957122050 2024-03-29T06:02:50.659 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-377957122050 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 377957122050' 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: waiting osd.3 seq 377957122050 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:02:50.660 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:02:50.661 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:02:51.319 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:02:51.319 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:163799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 163797 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:02:51.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:02:51.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:02:51.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:02:51.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:02:51.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:02:51.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:02:51.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:02:51.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:02:51.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:02:51.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:02:51.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:02:51.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:02:51.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:51.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:51.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:51.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:51.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:02:16.043902+0000 2024-03-29T06:02:51.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:02:52.272 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:02:52.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:02:16.043902+0000 2024-03-29T06:02:52.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:02:52.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:02:16.043902+0000 2024-03-29T06:02:52.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:02:52.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:02:52.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:52.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:52.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:52.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:52.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:52.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:52.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:02:16.043902+0000 '>' 2024-03-29T06:02:16.043902+0000 2024-03-29T06:02:52.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:02:53.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:02:53.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:53.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:53.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:53.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:53.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:53.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:53.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:02:16.043902+0000 '>' 2024-03-29T06:02:16.043902+0000 2024-03-29T06:02:53.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:02:54.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:02:54.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:54.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:54.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:54.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:54.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:54.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:55.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:02:16.043902+0000 '>' 2024-03-29T06:02:16.043902+0000 2024-03-29T06:02:55.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:02:56.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:02:56.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:02:56.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:02:56.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:02:56.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:02:56.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:02:56.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:02:53.167279+0000 '>' 2024-03-29T06:02:16.043902+0000 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 166668"' 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 166668' 2024-03-29T06:02:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 166669"' 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 166669' 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 166668 166669' 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 166668 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/166671: /' 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:02:56.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/166673: /' 2024-03-29T06:03:00.367 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.367 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:03:00.368 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: _ 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: hinfo_key 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: snapset 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:03:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:03:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:03:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:03:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:03:00.372 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:166671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:03:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:166671: start osd.3 2024-03-29T06:03:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:03:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:03:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:03:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:03:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.841 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: _ 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: hinfo_key 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: snapset 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:03:00.842 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:03:00.843 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:03:00.843 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:03:00.843 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:03:00.843 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:03:00.843 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:03:00.843 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:03:00.843 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:03:00.844 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:03:00.845 INFO:tasks.workunit.client.0.smithi123.stderr:166673: start osd.1 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 2024-03-29T06:03:00.404+0000 7f8efea6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 2024-03-29T06:03:00.408+0000 7f8efea6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 2024-03-29T06:03:00.408+0000 7f8efea6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:04.679 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 0 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 2024-03-29T06:03:00.976+0000 7f8efea6a7c0 -1 Falling back to public interface 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:04.680 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 1 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 2 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 2024-03-29T06:03:03.852+0000 7f8efea6a7c0 -1 osd.3 90 log_to_monitors true 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:04.681 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:04.682 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:03:04.682 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 3 2024-03-29T06:03:04.682 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:05.609 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-29T06:03:05.609 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 2024-03-29T06:03:00.868+0000 7f83652a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 2024-03-29T06:03:00.872+0000 7f83652a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 2024-03-29T06:03:00.872+0000 7f83652a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:05.610 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 0 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 2024-03-29T06:03:01.420+0000 7f83652a47c0 -1 Falling back to public interface 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 1 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:05.611 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:03:05.613 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 2 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 2024-03-29T06:03:04.516+0000 7f83652a47c0 -1 osd.1 90 log_to_monitors true 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:03:05.615 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 3 2024-03-29T06:03:05.615 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:05.615 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:03:06.220 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: osd.3 up in weight 1 up_from 95 up_thru 88 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6810/946341725,v1:127.0.0.1:6811/946341725] [v2:127.0.0.1:6812/946341725,v1:127.0.0.1:6813/946341725] exists,up e0b6f1e6-dd56-4e69-980c-e61911bf8222 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:03:06.221 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 1 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 2 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: 3' 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=2 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 2 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-2' 2024-03-29T06:03:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:07.712 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtesteph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:07.712 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:07.712 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:07.712 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:03:07.712 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 4 2024-03-29T06:03:07.712 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:07.712 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: osd.1 up in weight 1 up_from 98 up_thru 98 down_at 91 last_clean_interval [84,90) [v2:127.0.0.1:6826/58889857,v1:127.0.0.1:6827/58889857] [v2:127.0.0.1:6828/58889857,v1:127.0.0.1:6829/58889857] exists,up 78ac5e0c-e073-4220-872e-936bacad3c3d 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:03:07.713 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 1 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 2 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: 3' 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-03-29T06:03:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-03-29T06:03:07.715 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-03-29T06:03:07.715 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:07.738 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-h/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:03:07.738 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449101 2024-03-29T06:03:07.738 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449101 2024-03-29T06:03:07.738 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-2 2-335007449101' 2024-03-29T06:03:07.738 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-2 2-335007449101 3-408021893122' 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: waiting osd.0 seq 21474836547 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:03:07.739 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836547 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-2 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-2 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=2 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: waiting osd.1 seq 2 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 2 -lt 2 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449101 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:03:07.741 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449101 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/stelpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-420906795011' 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449102 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449102 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-420906795011 2-335007449102' 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:03:08.905 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-420906795011 2-335007449102 3-408021893123' 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: waiting osd.0 seq 21474836548 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:03:08.906 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836548 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-420906795011 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-420906795011 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 420906795011' 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: waiting osd.1 seq 420906795011 2024-03-29T06:03:08.907 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795011 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449102 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449102 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449102 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449102' 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: waiting osd.2 seq 335007449102 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:03:10.376 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449102 -lt 335007449102 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-408021893123 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-408021893123 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 408021893123' 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: waiting osd.3 seq 408021893123 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893123 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:03:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:03:10.378 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:03:10.378 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:03:10.378 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:03:10.378 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:03:10.378 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:03:10.378 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:03:10.378 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:03:10.474 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clandalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:10.474 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449101 2024-03-29T06:03:10.474 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449101' 2024-03-29T06:03:10.474 INFO:tasks.workunit.client.0.smithi123.stderr:166671: waiting osd.2 seq 335007449101 2024-03-29T06:03:10.474 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:03:10.474 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449100 -lt 335007449101 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449102 -lt 335007449101 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-408021893122 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-408021893122 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 408021893122' 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: waiting osd.3 seq 408021893122 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893122 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:03:10.475 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:03:10.476 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:03:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cean: cur_active_clean=5 2024-03-29T06:03:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:03:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:03:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:166673: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:03:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:03:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:03:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:166673: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:03:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:eph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:166671: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:166671: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 166669 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:03:10.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:03:10.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:03:10.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:03:10.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:03:10.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:03:10.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:03:10.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:03:10.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:03:10.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:03:10.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:03:10.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:03:10.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:03:11.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:03:11.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:03:11.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:03:11.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:03:11.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:03:11.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:03:11.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:03:11.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:03:11.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:03:11.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:03:11.212 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:03:11.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:03:11.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:03:11.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:03:11.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:03:11.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:11.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:11.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:03:11.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:03:11.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:03:11.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:03:11.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:03:11.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:03:11.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:03:11.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:03:11.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:03:11.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:03:11.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:03:11.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:03:11.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:03:11.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:03:11.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:03:11.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:03:11.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:03:11.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:03:11.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:03:11.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:03:11.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:11.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:11.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:03:11.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:03:11.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:03:11.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:03:11.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:03:11.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:11.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:11.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-29T06:03:11.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:03:11.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:03:11.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:03:11.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:03:11.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:03:11.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T06:03:11.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:03:11.675 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:03:11.675 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:03:11.675 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:03:11.675 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:11.676 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:11.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:11.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T06:03:11.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:03:11.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:03:11.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:03:11.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:03:11.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:03:11.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:03:11.889 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:03:11.890 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:03:12.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:03:12.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:03:12.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:03:12.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:03:12.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:03:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:03:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:03:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:03:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:03:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:12.010 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:03:12.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:03:12.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:03:12.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-29T06:03:12.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:03:12.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:03:12.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:03:12.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:03:12.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:03:12.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:03:12.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:03:12.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:03:12.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:03:12.994 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:03:12.994 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:12.994 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:12.994 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:12.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:03:12.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:03:13.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:03:13.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:03:13.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:03:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50b469a0-f4d9-4db5-a814-482673abea20 2024-03-29T06:03:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50b469a0-f4d9-4db5-a814-482673abea20' 2024-03-29T06:03:13.038 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 50b469a0-f4d9-4db5-a814-482673abea20 2024-03-29T06:03:13.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:03:13.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChWQZm36REAxAAuKtF0UvJp+0CvWmzk2VGTQ== 2024-03-29T06:03:13.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChWQZm36REAxAAuKtF0UvJp+0CvWmzk2VGTQ=="}' 2024-03-29T06:03:13.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50b469a0-f4d9-4db5-a814-482673abea20 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:03:13.305 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:03:13.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:03:13.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChWQZm36REAxAAuKtF0UvJp+0CvWmzk2VGTQ== --osd-uuid 50b469a0-f4d9-4db5-a814-482673abea20 2024-03-29T06:03:13.357 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:13.355+0000 7fd6e3c767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:13.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:13.367+0000 7fd6e3c767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:13.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:13.371+0000 7fd6e3c767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:13.374 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:13.371+0000 7fd6e3c767c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:03:13.374 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:13.371+0000 7fd6e3c767c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:03:13.374 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:13.371+0000 7fd6e3c767c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:03:13.472 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:13.467+0000 7fd6e3c767c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:03:19.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:03:19.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:03:19.883 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:03:19.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:03:19.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:03:20.260 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:03:20.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:03:20.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:03:20.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:03:20.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:03:20.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:03:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:20.287+0000 7f677ad4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:20.294 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:20.291+0000 7f677ad4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:20.296 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:20.291+0000 7f677ad4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:20.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:03:20.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:03:20.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:03:20.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:03:20.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:03:20.604 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:03:20.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:20.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:03:20.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:20.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:20.850 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:20.847+0000 7f677ad4d7c0 -1 Falling back to public interface 2024-03-29T06:03:20.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:21.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:21.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:21.945 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:03:21.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:03:21.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:21.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:22.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:23.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:23.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:23.249 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:03:23.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:03:23.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:23.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:23.439 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:23.435+0000 7f677ad4d7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:03:23.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:24.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:24.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:24.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:03:24.554 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:03:24.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:24.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:24.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:25.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:25.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:25.874 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:03:25.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:03:25.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:25.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:26.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:27.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:27.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:27.336 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:03:27.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:03:27.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:27.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:27.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:28.648 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:03:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:03:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:28.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:29.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:29.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:29.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:03:29.967 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:03:29.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:29.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:30.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:31.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:31.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:31.307 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:03:31.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:03:31.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:31.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:31.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:32.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:32.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:32.626 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:03:32.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:03:32.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:32.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:32.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:33.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:33.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:33.940 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:03:33.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:03:33.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:33.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:34.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:35.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:35.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:35.324 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:03:35.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:03:35.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:35.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:35.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:36.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:36.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:36.650 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:03:36.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:03:36.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:36.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:36.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:37.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:37.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:37.971 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:03:37.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:03:37.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:37.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:38.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:39.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:39.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:39.534 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:03:39.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:03:39.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:39.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:40.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:41.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:41.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:41.461 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:03:41.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:03:41.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:41.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:42.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:43.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:43.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:43.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:03:43.264 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:03:43.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:43.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:44.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:45.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:45.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:45.240 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:03:45.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:03:45.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:45.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:46.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:47.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:47.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:47.176 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:03:47.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:03:47.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:47.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:47.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:48.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:48.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:48.771 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:03:48.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:03:48.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:48.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:03:49.086 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3739255617,v1:127.0.0.1:6803/3739255617] [v2:127.0.0.1:6804/3739255617,v1:127.0.0.1:6805/3739255617] exists,up 50b469a0-f4d9-4db5-a814-482673abea20 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:03:49.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:03:49.089 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:03:49.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:49.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:03:49.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:03:49.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:03:49.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9d702115-ae14-4838-8542-30bdcb55a5a6' 2024-03-29T06:03:49.093 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:03:49.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:03:49.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFWQZmJ/vHBhAAD3oQ25gH94kTR7R0lO0lnA== 2024-03-29T06:03:49.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFWQZmJ/vHBhAAD3oQ25gH94kTR7R0lO0lnA=="}' 2024-03-29T06:03:49.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9d702115-ae14-4838-8542-30bdcb55a5a6 -i td/osd-scrub-repair/1/new.json 2024-03-29T06:03:49.507 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:03:49.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:03:49.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFWQZmJ/vHBhAAD3oQ25gH94kTR7R0lO0lnA== --osd-uuid 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:03:49.561 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:49.558+0000 7f4e479447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:49.618 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:49.614+0000 7f4e479447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:49.620 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:49.614+0000 7f4e479447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:49.621 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:49.618+0000 7f4e479447c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:03:49.621 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:49.618+0000 7f4e479447c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:03:49.621 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:49.618+0000 7f4e479447c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:03:49.702 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:49.698+0000 7f4e479447c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:03:56.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:03:56.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:03:56.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:03:56.705 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:03:56.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:03:57.081 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:03:57.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:03:57.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:03:57.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:03:57.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:03:57.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:03:57.114 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:57.114+0000 7f1b309057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:57.147 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:57.146+0000 7f1b309057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:57.148 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:57.146+0000 7f1b309057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:03:57.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:03:57.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:03:57.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:03:57.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:03:57.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:03:57.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:57.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:03:57.422 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:03:57.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:57.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:03:57.703 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:03:57.702+0000 7f1b309057c0 -1 Falling back to public interface 2024-03-29T06:03:57.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:03:58.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:03:58.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:03:58.801 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:03:58.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:03:58.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:03:58.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:03:59.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:00.102 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:04:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:04:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:00.137 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:00.138+0000 7f1b309057c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:04:00.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:01.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:01.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:01.599 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:04:01.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:04:01.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:01.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:01.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:02.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:02.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:02.947 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:04:02.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:04:02.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:02.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:03.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:04.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:04.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:04.280 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:04:04.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:04:04.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:04.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:04.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:05.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:05.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:05.596 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:04:05.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:04:05.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:05.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:05.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:06.911 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:04:06.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:04:06.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:06.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:07.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:08.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:08.298 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:04:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:04:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:08.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:09.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:09.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:09.634 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:04:09.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:04:09.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:09.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:09.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:10.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:10.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:10.946 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:04:10.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:04:10.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:10.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:11.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:12.316 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:04:12.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:04:12.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:12.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:12.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:13.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:13.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:13.634 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:04:13.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:04:13.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:13.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:13.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:14.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:14.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:14.945 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:04:14.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:04:14.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:14.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:15.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:16.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:16.664 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:04:16.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:16.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:04:16.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:16.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:17.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:18.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:18.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:18.280 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:04:18.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:04:18.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:18.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:18.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:19.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:19.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:19.642 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:04:19.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:04:19.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:19.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:20.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:21.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:21.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:21.127 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:04:21.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:04:21.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:21.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:21.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:22.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:22.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:22.756 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:04:22.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:04:22.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:22.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:23.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:24.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:24.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:24.075 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:04:24.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:04:24.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:24.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:24.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:25.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:25.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:25.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:04:25.680 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:04:25.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:25.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:26.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:27.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:27.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:27.003 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:04:27.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:04:27.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:27.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:04:27.328 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/934792196,v1:127.0.0.1:6811/934792196] [v2:127.0.0.1:6812/934792196,v1:127.0.0.1:6813/934792196] exists,up 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:04:27.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:04:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:04:27.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:04:27.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:04:27.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bdf43b7d-4183-4650-a893-494f090724c4 2024-03-29T06:04:27.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bdf43b7d-4183-4650-a893-494f090724c4' 2024-03-29T06:04:27.335 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 bdf43b7d-4183-4650-a893-494f090724c4 2024-03-29T06:04:27.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:04:27.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrWQZmjnglFRAAYXAT4Ar+NMh+lk3o6OAOYA== 2024-03-29T06:04:27.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrWQZmjnglFRAAYXAT4Ar+NMh+lk3o6OAOYA=="}' 2024-03-29T06:04:27.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bdf43b7d-4183-4650-a893-494f090724c4 -i td/osd-scrub-repair/2/new.json 2024-03-29T06:04:27.744 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:04:27.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T06:04:27.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrWQZmjnglFRAAYXAT4Ar+NMh+lk3o6OAOYA== --osd-uuid bdf43b7d-4183-4650-a893-494f090724c4 2024-03-29T06:04:27.802 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:27.801+0000 7fe00d9d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:04:27.854 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:27.853+0000 7fe00d9d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:04:27.855 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:27.853+0000 7fe00d9d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:04:27.856 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:27.853+0000 7fe00d9d17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:04:27.856 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:27.853+0000 7fe00d9d17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:04:27.856 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:27.853+0000 7fe00d9d17c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:04:27.924 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:27.921+0000 7fe00d9d17c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T06:04:35.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T06:04:35.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:04:35.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T06:04:35.021 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T06:04:35.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:04:35.414 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T06:04:35.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T06:04:35.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:04:35.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:04:35.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:04:35.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:04:35.447 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:35.445+0000 7fa6a4df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:04:35.474 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:35.473+0000 7fa6a4df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:04:35.475 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:35.473+0000 7fa6a4df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:04:35.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T06:04:35.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:04:35.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:04:35.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:04:35.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:04:35.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:35.752 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:04:35.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:04:35.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:35.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:36.030 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:36.029+0000 7fa6a4df87c0 -1 Falling back to public interface 2024-03-29T06:04:36.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:37.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:37.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:37.066 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:04:37.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:04:37.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:37.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:37.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:38.376 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:04:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:04:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:38.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:38.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:38.746 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:04:38.745+0000 7fa6a4df87c0 -1 osd.2 0 log_to_monitors true 2024-03-29T06:04:39.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:39.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:39.678 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:04:39.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:04:39.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:39.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:40.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:41.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:41.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:41.111 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:04:41.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:04:41.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:41.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:42.428 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:04:42.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:04:42.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:42.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:42.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:43.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:43.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:43.771 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:04:43.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:04:43.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:43.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:44.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:45.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:45.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:45.095 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:04:45.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:04:45.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:45.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:45.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:46.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:46.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:46.416 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:04:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:04:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:46.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:47.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:47.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:47.822 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:04:47.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:04:47.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:47.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:48.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:49.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:49.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:49.180 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:04:49.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:04:49.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:49.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:49.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:50.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:50.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:50.505 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:04:50.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:04:50.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:50.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:50.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:51.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:04:51.821 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:04:51.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:51.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:52.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:53.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:53.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:53.229 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:04:53.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:04:53.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:53.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:53.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:54.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:54.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:54.726 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:04:54.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:04:54.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:54.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:55.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:56.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:56.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:56.385 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:04:56.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:04:56.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:56.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:56.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:57.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:57.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:57.821 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:04:57.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:04:57.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:57.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:04:58.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:04:59.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:04:59.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:04:59.587 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:04:59.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:04:59.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:04:59.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:05:00.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:01.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:01.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:01.105 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:05:01.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:05:01.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:01.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:05:01.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:02.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:02.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:02.658 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:05:02.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:05:02.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:02.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:05:03.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:04.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:04.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:04.074 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:05:04.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:05:04.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:04.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:05:04.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:05.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:05.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:05.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:05:05.412 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:05:05.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:05.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:05:05.739 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1365349986,v1:127.0.0.1:6819/1365349986] [v2:127.0.0.1:6820/1365349986,v1:127.0.0.1:6821/1365349986] exists,up bdf43b7d-4183-4650-a893-494f090724c4 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:05:05.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:05:05.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:05:05.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:05:05.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:05:05.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:05:05.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6587b6c5-5bfe-48b7-8cc9-cb634c382244' 2024-03-29T06:05:05.746 INFO:tasks.workunit.client.0.smithi123.stdout:add osd3 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:05:05.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:05:05.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARWgZmcyu2LRAAihMX5Cs9DZalgiiv0yXlnw== 2024-03-29T06:05:05.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARWgZmcyu2LRAAihMX5Cs9DZalgiiv0yXlnw=="}' 2024-03-29T06:05:05.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6587b6c5-5bfe-48b7-8cc9-cb634c382244 -i td/osd-scrub-repair/3/new.json 2024-03-29T06:05:06.145 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:05:06.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-29T06:05:06.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARWgZmcyu2LRAAihMX5Cs9DZalgiiv0yXlnw== --osd-uuid 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:05:06.196 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:06.192+0000 7f8c808d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:05:06.255 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:06.252+0000 7f8c808d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:05:06.256 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:06.252+0000 7f8c808d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:05:06.257 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:06.256+0000 7f8c808d67c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:05:06.257 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:06.256+0000 7f8c808d67c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:05:06.257 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:06.256+0000 7f8c808d67c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:05:06.338 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:06.336+0000 7f8c808d67c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-29T06:05:13.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-29T06:05:13.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:05:13.435 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd3 key to auth repository 2024-03-29T06:05:13.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-29T06:05:13.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:05:13.816 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.3 2024-03-29T06:05:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-29T06:05:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:05:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:05:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:05:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:05:13.848 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:13.844+0000 7fd404ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:05:13.850 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:13.848+0000 7fd404ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:05:13.851 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:13.848+0000 7fd404ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:05:14.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-29T06:05:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:05:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:05:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:05:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:05:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:14.098 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:05:14.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:05:14.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:14.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:14.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:14.427 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:14.424+0000 7fd404ce37c0 -1 Falling back to public interface 2024-03-29T06:05:15.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:15.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:15.410 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:05:15.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:05:15.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:15.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:15.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:16.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:16.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:16.728 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:05:16.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:05:16.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:16.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:17.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:17.114 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:05:17.112+0000 7fd404ce37c0 -1 osd.3 0 log_to_monitors true 2024-03-29T06:05:18.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:18.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:18.028 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:05:18.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:05:18.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:18.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:18.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:19.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:19.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:19.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:05:19.349 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:05:19.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:19.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:19.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:20.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:20.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:20.720 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:05:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:05:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:21.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:22.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:22.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:22.137 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:05:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:05:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:22.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:23.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:23.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:23.525 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:05:23.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:05:23.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:23.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:23.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:24.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:24.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:24.877 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:05:24.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:05:24.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:24.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:25.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:26.323 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:05:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:05:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:26.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:26.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:27.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:27.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:27.702 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:05:27.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:05:27.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:27.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:28.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:29.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:05:29.190 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:05:29.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:29.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:29.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:30.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:30.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:30.509 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:05:30.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:05:30.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:30.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:30.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:31.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:31.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:31.906 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:05:31.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:05:31.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:31.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:32.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:33.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:33.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:33.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:05:33.563 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:05:33.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:33.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:34.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:35.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:35.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:35.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:05:35.405 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:05:35.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:35.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:36.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:37.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:37.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:37.027 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:05:37.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:05:37.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:37.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:37.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:38.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:38.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:38.829 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:05:38.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:05:38.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:38.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:39.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:40.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:40.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:40.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:05:40.442 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:05:40.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:40.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:40.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:41.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:41.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:41.851 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:05:41.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:05:41.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:41.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:42.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:43.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:43.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:43.492 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:05:43.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:05:43.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:43.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:43.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:05:44.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:05:44.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:05:44.930 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:05:44.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:05:44.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:05:44.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:05:45.246 INFO:tasks.workunit.client.0.smithi123.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1951931942,v1:127.0.0.1:6827/1951931942] [v2:127.0.0.1:6828/1951931942,v1:127.0.0.1:6829/1951931942] exists,up 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:05:45.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:05:45.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:05:45.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:05:45.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-29T06:05:45.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:05:45.536 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:05:45.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:05:45.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:05:45.987 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:05:46.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:05:47.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:05:49.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-29T06:05:49.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:05:49.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:05:49.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:05:49.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:05:49.745 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:05:49.745 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:05:49.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:05:49.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:05:49.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:05:49.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:05:49.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:05:49.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:05:49.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:05:49.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:05:49.876 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:05:50.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:05:50.237 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:05:50.237 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:05:50.237 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:05:50.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:05:50.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:50.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:05:50.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-29T06:05:50.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-29T06:05:50.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-29T06:05:50.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:50.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:05:50.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-29T06:05:50.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-29T06:05:50.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672983' 2024-03-29T06:05:50.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:50.567 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:05:50.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-29T06:05:50.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-29T06:05:50.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509458' 2024-03-29T06:05:50.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:50.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:05:50.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-03-29T06:05:50.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-03-29T06:05:50.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509458 3-85899345931' 2024-03-29T06:05:50.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:05:50.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-29T06:05:50.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:05:50.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:05:50.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-29T06:05:50.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:05:50.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-29T06:05:50.845 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836514 2024-03-29T06:05:50.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-29T06:05:50.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:05:51.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-03-29T06:05:51.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:05:52.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:05:52.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:05:52.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-03-29T06:05:52.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:05:52.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-29T06:05:52.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:05:52.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:05:52.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-29T06:05:52.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:05:52.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-29T06:05:52.460 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672983 2024-03-29T06:05:52.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-29T06:05:52.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:05:52.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-29T06:05:52.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:05:52.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-29T06:05:52.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:05:52.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:05:52.805 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-29T06:05:52.805 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:05:52.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-29T06:05:52.806 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509458 2024-03-29T06:05:52.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-29T06:05:52.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:05:53.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-29T06:05:53.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:05:53.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-03-29T06:05:53.141 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:05:53.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:05:53.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-03-29T06:05:53.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:05:53.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-03-29T06:05:53.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-03-29T06:05:53.144 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345931 2024-03-29T06:05:53.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:05:53.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-03-29T06:05:53.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:05:53.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:05:53.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:05:53.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:05:53.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:05:53.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:05:53.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:05:53.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:05:53.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:05:53.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:05:53.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:05:54.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:05:54.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:05:54.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:05:54.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-29T06:05:54.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T06:05:54.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-29T06:05:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T06:05:55.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T06:05:55.803 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T06:05:55.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:05:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-29T06:05:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-29T06:05:57.250 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 2 allow_ec_overwrites to true 2024-03-29T06:05:57.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T06:05:57.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:05:57.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:05:57.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:05:57.269 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:05:57.270 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:05:57.270 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:05:57.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:05:57.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:05:57.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:05:57.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:05:57.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:05:57.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:05:57.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:05:57.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:05:57.376 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:05:57.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:05:57.797 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:05:57.797 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:05:57.797 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:05:57.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:05:57.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:57.797 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:05:57.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-03-29T06:05:57.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-03-29T06:05:57.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-03-29T06:05:57.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:57.949 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:05:58.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-29T06:05:58.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-29T06:05:58.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672986' 2024-03-29T06:05:58.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:58.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:05:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-29T06:05:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-29T06:05:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672986 2-64424509460' 2024-03-29T06:05:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:05:58.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:05:58.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345933 2024-03-29T06:05:58.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345933 2024-03-29T06:05:58.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672986 2-64424509460 3-85899345933' 2024-03-29T06:05:58.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:05:58.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-03-29T06:05:58.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:05:58.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:05:58.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-03-29T06:05:58.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:05:58.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-03-29T06:05:58.481 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836516 2024-03-29T06:05:58.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-03-29T06:05:58.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:05:58.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836516 2024-03-29T06:05:58.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:05:59.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:05:59.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:06:00.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836516 2024-03-29T06:06:00.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:00.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-29T06:06:00.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:00.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:06:00.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-29T06:06:00.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:00.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-29T06:06:00.105 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672986 2024-03-29T06:06:00.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-29T06:06:00.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:06:00.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-03-29T06:06:00.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:00.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-29T06:06:00.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:00.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:06:00.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-29T06:06:00.421 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:00.422 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509460 2024-03-29T06:06:00.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-29T06:06:00.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-29T06:06:00.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:06:00.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-29T06:06:00.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:00.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345933 2024-03-29T06:06:00.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:00.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:06:00.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345933 2024-03-29T06:06:00.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:00.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345933 2024-03-29T06:06:00.749 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345933 2024-03-29T06:06:00.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-03-29T06:06:00.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:06:01.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345933 -lt 85899345933 2024-03-29T06:06:01.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:06:01.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:01.086 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:01.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:06:01.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:06:01.497 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:06:01.497 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:06:01.497 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:06:01.497 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:06:01.497 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:06:01.497 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:06:01.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:06:01.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:06:01.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:01.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:06:02.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-29T06:06:02.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:06:02.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:06:02.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:06:02.662 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:06:02.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:06:03.089 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:06:03.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:06:03.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:06:03.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T06:06:03.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-29T06:06:03.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T06:06:03.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T06:06:03.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:06:03.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:06:03.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-29T06:06:03.966 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-29T06:06:03.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-29T06:06:03.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T06:06:03.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-29T06:06:03.967 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:06:03.967 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:06:04.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-29T06:06:04.313 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:04.313 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:06:04.313 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:06:04.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:06:04.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:06:04.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:06:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:06:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:06:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:06:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:06:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:06:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:06:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:06:04.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:06:06.515 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:06:07.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:06:07.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:06:07.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:06:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:06:07.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:06:07.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:06:07.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:06:07.066 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:06:07.066 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:06:07.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:06:07.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:06:07.068 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:06:07.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:06:07.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:06:07.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:06:07.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:06:07.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:06:07.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:06:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:07.095+0000 7f554ef667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:07.103 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:07.099+0000 7f554ef667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:07.105 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:07.099+0000 7f554ef667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:07.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:07.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:07.652 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:07.647+0000 7f554ef667c0 -1 Falling back to public interface 2024-03-29T06:06:07.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:08.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:08.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:08.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:06:08.796 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:08.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:08.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:09.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:10.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:10.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:10.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:06:10.113 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:06:10.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:10.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:10.439 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:10.434+0000 7f554ef667c0 -1 osd.3 37 log_to_monitors true 2024-03-29T06:06:10.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:11.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:11.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:11.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:06:11.556 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:06:11.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:11.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/2278733520,v1:127.0.0.1:6827/2278733520] [v2:127.0.0.1:6828/2278733520,v1:127.0.0.1:6829/2278733520] exists,up 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:06:12.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:06:12.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:06:12.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:06:12.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:06:12.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:06:12.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:06:12.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:06:12.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:06:12.011 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:06:12.011 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:06:12.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:06:12.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:06:12.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:06:12.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:06:12.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:06:12.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:06:12.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:06:12.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:06:12.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:06:12.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:06:12.539 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:12.539 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:06:12.539 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:06:12.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:06:12.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:12.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:06:12.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-03-29T06:06:12.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-03-29T06:06:12.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-03-29T06:06:12.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:12.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:06:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-03-29T06:06:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-03-29T06:06:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672990' 2024-03-29T06:06:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:12.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:06:12.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-29T06:06:12.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-29T06:06:12.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672990 2-64424509464' 2024-03-29T06:06:12.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:12.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:06:13.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-03-29T06:06:13.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-03-29T06:06:13.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672990 2-64424509464 3-176093659138' 2024-03-29T06:06:13.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:13.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-03-29T06:06:13.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:13.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:06:13.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-03-29T06:06:13.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:13.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-03-29T06:06:13.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-03-29T06:06:13.114 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836520 2024-03-29T06:06:13.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:06:13.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-03-29T06:06:13.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:06:14.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:06:14.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:06:14.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-03-29T06:06:14.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:14.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-03-29T06:06:14.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:14.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:06:14.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-03-29T06:06:14.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:14.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-03-29T06:06:14.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-03-29T06:06:14.745 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672990 2024-03-29T06:06:14.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:06:15.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672990 2024-03-29T06:06:15.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:15.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:15.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-29T06:06:15.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:06:15.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:15.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-29T06:06:15.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-29T06:06:15.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-29T06:06:15.091 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509464 2024-03-29T06:06:15.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:06:15.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-29T06:06:15.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:15.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:15.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-176093659138 2024-03-29T06:06:15.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:06:15.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-176093659138 2024-03-29T06:06:15.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:15.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-03-29T06:06:15.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-03-29T06:06:15.432 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 176093659138 2024-03-29T06:06:15.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:06:15.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-03-29T06:06:15.740 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:06:15.740 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:15.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:16.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:06:16.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:06:16.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:06:16.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:06:16.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:06:16.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:06:16.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:06:16.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:06:16.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:06:16.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:06:16.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:16.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:06:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:06:16.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:06:16.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:06:16.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:06:16.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:06:16.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:06:16.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:06:17.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:06:17.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:06:17.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:06:17.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:06:17.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:17.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:17.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:17.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:17.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:05:55.645123+0000 2024-03-29T06:06:17.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:06:17.711 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:06:17.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:05:55.645123+0000 2024-03-29T06:06:17.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:06:17.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:05:55.645123+0000 2024-03-29T06:06:17.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:06:17.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:06:17.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:17.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:17.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:17.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:17.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:17.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:18.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:05:55.645123+0000 '>' 2024-03-29T06:05:55.645123+0000 2024-03-29T06:06:18.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:19.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:19.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:19.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:19.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:19.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:19.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:19.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:19.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:05:55.645123+0000 '>' 2024-03-29T06:05:55.645123+0000 2024-03-29T06:06:19.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:20.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:20.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:20.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:20.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:20.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:20.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:20.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:20.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:05:55.645123+0000 '>' 2024-03-29T06:05:55.645123+0000 2024-03-29T06:06:20.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:21.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:21.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:21.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:21.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:21.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:21.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:21.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:21.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:18.435795+0000 '>' 2024-03-29T06:05:55.645123+0000 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:06:21.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:06:21.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:06:21.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:06:21.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:06:21.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:06:21.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:06:21.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:06:21.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:06:22.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:06:24.157 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:06:24.158 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:06:24.158 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:06:24.158 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:06:24.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:06:24.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:06:24.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:06:24.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:06:24.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:06:24.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:06:24.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:06:24.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:06:24.455 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:06:24.455 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:06:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:06:24.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:06:24.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:06:24.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:06:24.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:06:24.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:06:24.489 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:24.490+0000 7ff70be327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:24.492 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:24.490+0000 7ff70be327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:24.493 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:24.494+0000 7ff70be327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:06:24.777 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:06:24.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:24.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:25.031 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:25.030+0000 7ff70be327c0 -1 Falling back to public interface 2024-03-29T06:06:25.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:06:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:26.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:26.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:26.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:27.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:27.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:27.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:06:27.423 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:06:27.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:27.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:27.594 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:27.594+0000 7ff70be327c0 -1 osd.3 43 log_to_monitors true 2024-03-29T06:06:27.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:28.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:28.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:28.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:06:28.737 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:06:28.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:28.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 47 up_thru 47 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/3107356735,v1:127.0.0.1:6827/3107356735] [v2:127.0.0.1:6828/3107356735,v1:127.0.0.1:6829/3107356735] exists,up 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:06:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:06:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:06:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:06:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:06:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:06:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:06:29.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:06:29.134 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:06:29.134 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:06:29.134 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:06:29.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:06:29.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:06:29.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:06:29.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:06:29.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:06:29.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:06:29.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:06:29.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:06:29.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:06:29.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:06:29.560 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:29.560 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:06:29.560 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:06:29.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:06:29.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:29.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:06:29.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-03-29T06:06:29.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-03-29T06:06:29.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-03-29T06:06:29.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:29.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:06:29.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672994 2024-03-29T06:06:29.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672994 2024-03-29T06:06:29.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949672994' 2024-03-29T06:06:29.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:29.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:06:30.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-03-29T06:06:30.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-03-29T06:06:30.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949672994 2-64424509468' 2024-03-29T06:06:30.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:30.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:06:30.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462914 2024-03-29T06:06:30.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462914 2024-03-29T06:06:30.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949672994 2-64424509468 3-201863462914' 2024-03-29T06:06:30.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:30.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-03-29T06:06:30.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:30.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:06:30.377 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-03-29T06:06:30.377 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:30.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-03-29T06:06:30.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-03-29T06:06:30.378 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836525 2024-03-29T06:06:30.379 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:06:30.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836525 2024-03-29T06:06:30.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:06:31.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:06:31.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:06:31.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-03-29T06:06:31.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:31.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672994 2024-03-29T06:06:31.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:31.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:06:31.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672994 2024-03-29T06:06:31.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:31.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672994 2024-03-29T06:06:31.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-03-29T06:06:31.986 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672994 2024-03-29T06:06:31.986 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:06:32.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672994 -lt 42949672994 2024-03-29T06:06:32.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:32.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-03-29T06:06:32.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:32.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:06:32.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-03-29T06:06:32.320 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:32.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-03-29T06:06:32.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-03-29T06:06:32.321 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509468 2024-03-29T06:06:32.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:06:32.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509469 -lt 64424509468 2024-03-29T06:06:32.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:32.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462914 2024-03-29T06:06:32.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:32.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:06:32.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462914 2024-03-29T06:06:32.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:32.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462914 2024-03-29T06:06:32.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462914' 2024-03-29T06:06:32.655 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 201863462914 2024-03-29T06:06:32.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:06:33.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462914 -lt 201863462914 2024-03-29T06:06:33.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:06:33.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:33.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:33.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:06:33.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:06:33.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:06:33.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:06:33.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:06:33.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:06:33.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:06:33.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:06:33.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:06:33.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:06:33.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:33.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:34.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:06:34.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:06:34.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:06:34.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:06:34.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:06:34.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-29T06:06:34.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:06:34.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:06:34.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-29T06:06:34.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:06:34.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:06:34.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:06:34.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:06:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T06:06:36.740 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:06:37.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:06:37.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:06:37.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:06:37.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:06:37.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:06:37.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:06:37.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:06:37.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:06:37.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:06:37.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:06:37.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:06:37.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:06:37.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:06:37.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:06:37.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:06:37.278 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:06:37.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:06:37.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:06:37.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:06:37.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:06:37.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:06:37.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:06:37.312 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:37.310+0000 7f9acb2967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:37.315 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:37.314+0000 7f9acb2967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:37.317 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:37.314+0000 7f9acb2967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:37.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:06:37.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:37.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:06:37.855 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:37.854+0000 7f9acb2967c0 -1 Falling back to public interface 2024-03-29T06:06:37.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:38.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:38.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:38.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:06:38.907 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:38.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:06:39.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:40.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:40.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:40.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:06:40.214 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:06:40.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:40.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:06:40.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:40.576 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:40.574+0000 7f9acb2967c0 -1 osd.1 49 log_to_monitors true 2024-03-29T06:06:41.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:41.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:41.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:06:41.534 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:06:41.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:41.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 50 last_clean_interval [10,49) [v2:127.0.0.1:6810/3136739908,v1:127.0.0.1:6811/3136739908] [v2:127.0.0.1:6812/3136739908,v1:127.0.0.1:6813/3136739908] exists,up 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:06:41.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:06:41.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:06:41.858 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:06:41.858 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:06:41.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:06:41.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:06:41.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:06:41.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:06:41.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:06:41.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:06:41.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:06:41.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:06:41.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:06:42.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:06:42.553 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:42.553 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:06:42.553 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:06:42.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:06:42.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:42.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:06:42.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836528 2024-03-29T06:06:42.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836528 2024-03-29T06:06:42.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528' 2024-03-29T06:06:42.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:42.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:06:42.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233986 2024-03-29T06:06:42.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233986 2024-03-29T06:06:42.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-231928233986' 2024-03-29T06:06:42.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:42.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:06:43.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509472 2024-03-29T06:06:43.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509472 2024-03-29T06:06:43.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-231928233986 2-64424509472' 2024-03-29T06:06:43.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:06:43.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:06:43.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462918 2024-03-29T06:06:43.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462918 2024-03-29T06:06:43.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-231928233986 2-64424509472 3-201863462918' 2024-03-29T06:06:43.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:43.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836528 2024-03-29T06:06:43.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:43.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:06:43.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836528 2024-03-29T06:06:43.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:43.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836528 2024-03-29T06:06:43.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-03-29T06:06:43.243 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836528 2024-03-29T06:06:43.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:06:43.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836528 2024-03-29T06:06:43.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:06:44.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:06:44.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:06:44.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836528 2024-03-29T06:06:44.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:44.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233986 2024-03-29T06:06:44.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:44.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:06:44.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233986 2024-03-29T06:06:44.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:44.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233986 2024-03-29T06:06:44.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2024-03-29T06:06:44.872 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 231928233986 2024-03-29T06:06:44.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:06:45.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233986 -lt 231928233986 2024-03-29T06:06:45.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:45.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509472 2024-03-29T06:06:45.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:45.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:06:45.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509472 2024-03-29T06:06:45.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:45.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509472 2024-03-29T06:06:45.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-03-29T06:06:45.200 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509472 2024-03-29T06:06:45.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:06:45.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509472 -lt 64424509472 2024-03-29T06:06:45.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:06:45.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462918 2024-03-29T06:06:45.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:06:45.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:06:45.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462918 2024-03-29T06:06:45.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:06:45.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462918 2024-03-29T06:06:45.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462918' 2024-03-29T06:06:45.535 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 201863462918 2024-03-29T06:06:45.535 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:06:45.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462918 -lt 201863462918 2024-03-29T06:06:45.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:06:45.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:45.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:46.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:06:46.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:06:46.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:06:46.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:06:46.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:06:46.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:06:46.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:06:46.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:06:46.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:06:46.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:06:46.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:06:46.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:06:46.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:06:46.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:06:46.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:06:46.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:06:46.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:06:46.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:06:46.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:06:46.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:06:47.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:06:47.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:06:47.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:06:47.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:06:47.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:47.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:47.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:47.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:47.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:47.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:06:47.955 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:06:47.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:47.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:06:47.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:47.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:06:47.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:06:47.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:47.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:47.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:47.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:47.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:47.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:48.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:18.435795+0000 '>' 2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:48.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:49.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:49.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:49.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:49.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:49.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:49.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:49.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:49.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:18.435795+0000 '>' 2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:49.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:50.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:50.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:50.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:50.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:50.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:50.601 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:50.601 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:50.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:18.435795+0000 '>' 2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:50.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:51.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:51.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:51.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:51.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:51.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:51.922 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:51.922 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:52.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:18.435795+0000 '>' 2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:52.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:53.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:53.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:53.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:53.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:53.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:53.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:53.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:53.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:18.435795+0000 '>' 2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:53.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:06:54.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:06:54.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:06:54.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:06:54.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:06:54.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:06:54.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:06:54.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:06:54.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:48.630482+0000 '>' 2024-03-29T06:06:18.435795+0000 2024-03-29T06:06:54.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:06:54.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:06:54.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:06:54.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:06:54.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:06:54.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:06:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:06:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:06:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:06:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:06:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:06:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:06:54.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:06:54.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:06:54.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:06:54.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:06:54.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:06:55.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:06:55.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:06:55.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:06:55.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:06:55.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:06:55.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:06:55.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:06:55.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:06:56.902 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:06:56.903 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:06:56.903 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:06:56.903 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:06:57.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:06:57.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:06:57.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:57.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:06:57.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:06:57.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:06:57.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:06:57.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:06:57.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:06:57.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:06:57.483 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:57.481+0000 7f7834fc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:57.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:57.485+0000 7f7834fc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:57.488 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:57.485+0000 7f7834fc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:06:57.763 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:06:57.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:57.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:58.023 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:06:58.021+0000 7f7834fc17c0 -1 Falling back to public interface 2024-03-29T06:06:58.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:06:59.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:06:59.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:06:59.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:06:59.069 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:06:59.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:06:59.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:06:59.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:00.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:00.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:00.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:07:00.380 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:07:00.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:00.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:07:00.543 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:07:00.541+0000 7f7834fc17c0 -1 osd.3 56 log_to_monitors true 2024-03-29T06:07:00.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:01.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:01.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:01.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:07:01.800 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:07:01.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:01.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:07:02.248 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [47,56) [v2:127.0.0.1:6826/1635571538,v1:127.0.0.1:6827/1635571538] [v2:127.0.0.1:6828/1635571538,v1:127.0.0.1:6829/1635571538] exists,up 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:07:02.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:07:02.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:07:02.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:07:02.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:07:02.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:07:02.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:07:02.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:07:02.560 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:07:02.560 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:07:02.560 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:07:02.560 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:07:02.560 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:07:02.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:07:02.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:07:02.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:07:02.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:07:02.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:07:02.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:07:02.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:07:02.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:07:02.779 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:07:02.780 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:07:02.780 INFO:tasks.workunit.client.0.smithi123.stderr:3' 2024-03-29T06:07:02.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:07:02.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:02.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:07:02.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836533 2024-03-29T06:07:02.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836533 2024-03-29T06:07:02.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533' 2024-03-29T06:07:02.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:02.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:07:03.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233991 2024-03-29T06:07:03.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233991 2024-03-29T06:07:03.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-231928233991' 2024-03-29T06:07:03.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:03.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:07:03.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509477 2024-03-29T06:07:03.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509477 2024-03-29T06:07:03.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-231928233991 2-64424509477' 2024-03-29T06:07:03.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:03.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:07:03.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-03-29T06:07:03.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-03-29T06:07:03.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-231928233991 2-64424509477 3-257698037762' 2024-03-29T06:07:03.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:03.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836533 2024-03-29T06:07:03.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:03.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:07:03.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836533 2024-03-29T06:07:03.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:03.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836533 2024-03-29T06:07:03.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-03-29T06:07:03.354 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836533 2024-03-29T06:07:03.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:03.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836533 2024-03-29T06:07:03.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:07:04.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:07:04.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:04.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836533 2024-03-29T06:07:04.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:04.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233991 2024-03-29T06:07:04.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:04.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:07:04.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233991 2024-03-29T06:07:04.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:04.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233991 2024-03-29T06:07:04.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233991' 2024-03-29T06:07:04.963 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 231928233991 2024-03-29T06:07:04.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:07:05.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233991 -lt 231928233991 2024-03-29T06:07:05.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:05.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509477 2024-03-29T06:07:05.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:05.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:07:05.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509477 2024-03-29T06:07:05.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:05.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509477 2024-03-29T06:07:05.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-03-29T06:07:05.307 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509477 2024-03-29T06:07:05.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:07:05.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509477 -lt 64424509477 2024-03-29T06:07:05.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:05.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037762 2024-03-29T06:07:05.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:05.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:07:05.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037762 2024-03-29T06:07:05.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:05.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-03-29T06:07:05.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-03-29T06:07:05.654 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 257698037762 2024-03-29T06:07:05.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:05.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-03-29T06:07:05.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:07:05.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:05.998 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:06.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:07:06.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:06.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:06.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:06.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:06.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:06.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:06.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:06.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:07:06.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:06.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:06.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:07.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:07:07.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:07:07.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:07:07.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:07:07.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:07:07.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-29T06:07:07.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:07:07.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 183560"' 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 183560' 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 183562"' 2024-03-29T06:07:07.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 183562' 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 183560 183562' 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 183560 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/183563: /' 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:07:07.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/183565: /' 2024-03-29T06:07:10.198 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:07:10.199 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:07:10.200 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:07:10.201 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:07:10.204 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:07:10.204 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:07:10.204 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:07:10.204 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:07:10.204 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:07:10.205 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: start osd.1 2024-03-29T06:07:10.206 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:07:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-29T06:07:10.370 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:07:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:07:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:07:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:07:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:07:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:07:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:07:10.371 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:07:10.372 INFO:tasks.workunit.client.0.smithi123.stderr:183565: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:07:10.373 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:07:10.374 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:07:10.374 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:07:10.374 INFO:tasks.workunit.client.0.smithi123.stderr:183565: start osd.2 2024-03-29T06:07:10.374 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:07:15.053 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 2024-03-29T06:07:10.397+0000 7f4a85da67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 2024-03-29T06:07:10.401+0000 7f4a85da67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 2024-03-29T06:07:10.401+0000 7f4a85da67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:07:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 0 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 2024-03-29T06:07:10.953+0000 7f4a85da67c0 -1 Falling back to public interface 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 1 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:15.055 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 2 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 2024-03-29T06:07:14.249+0000 7f4a85da67c0 -1 osd.2 62 log_to_monitors true 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 3 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 2024-03-29T06:07:10.237+0000 7f65220db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 2024-03-29T06:07:10.245+0000 7f65220db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 2024-03-29T06:07:10.249+0000 7f65220db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:15.057 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 0 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 2024-03-29T06:07:10.813+0000 7f65220db7c0 -1 Falling back to public interface 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 1 2024-03-29T06:07:15.058 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 2 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 2024-03-29T06:07:13.773+0000 7f65220db7c0 -1 osd.1 62 log_to_monitors true 2024-03-29T06:07:15.059 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:15.060 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:15.060 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:07:15.060 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 3 2024-03-29T06:07:15.060 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:15.060 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:16.390 INFO:tasks.workunit.client.0.smithi123.stderr:183563: osd.1 up in weight 1 up_from 67 up_thru 57 down_at 63 last_clean_interval [54,62) [v2:127.0.0.1:6810/2259024070,v1:127.0.0.1:6811/2259024070] [v2:127.0.0.1:6812/2259024070,v1:127.0.0.1:6813/2259024070] exists,up 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:07:16.390 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:07:16.390 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:07:16.390 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:07:16.390 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:07:16.391 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 1 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 2 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: 3' 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:16.392 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:07:16.393 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-29T06:07:16.393 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-29T06:07:16.393 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-287762808834' 2024-03-29T06:07:16.393 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:16.994 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph s.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:16.995 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:16.995 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:16.995 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:07:16.995 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 4 2024-03-29T06:07:16.995 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:16.995 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:16.996 INFO:tasks.workunit.client.0.smithi123.stderr:183565: osd.2 up in weight 1 up_from 68 up_thru 50 down_at 63 last_clean_interval [15,62) [v2:127.0.0.1:6818/1877504795,v1:127.0.0.1:6819/1877504795] [v2:127.0.0.1:6820/1877504795,v1:127.0.0.1:6821/1877504795] exists,up bdf43b7d-4183-4650-a893-494f090724c4 2024-03-29T06:07:16.996 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:07:16.996 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:07:16.996 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:07:16.996 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:07:16.996 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:07:16.997 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:07:16.997 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:07:16.997 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:07:16.997 INFO:tasks.workunit.client.0.smithi123.stderr:183565: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:07:16.997 INFO:tasks.workunit.client.0.smithi123.stderr:183565: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:07:16.997 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:07:16.998 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 1 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 2 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: 3' 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-03-29T06:07:16.999 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-287762808835' 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776131 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776131 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-287762808835 2-292057776131' 2024-03-29T06:07:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037767 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037767 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-287762808835 2-292057776131 3-257698037767' 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-03-29T06:07:18.259 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: waiting osd.0 seq 21474836538 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836538 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808835 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808835 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: waiting osd.1 seq 287762808835 2024-03-29T06:07:18.260 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221tell osd.2 flush_pg_stats 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-287762808834 2-292057776130' 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037766 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037766 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-287762808834 2-292057776130 3-257698037766' 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-03-29T06:07:18.911 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: waiting osd.0 seq 21474836537 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836537 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-03-29T06:07:18.912 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: waiting osd.1 seq 287762808834 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-03-29T06:07:18.913 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:20.309 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-292057776130 2024-03-29T06:07:20.309 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:20.309 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:07:20.309 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-292057776130 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 292057776130' 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: waiting osd.2 seq 292057776130 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776130 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037766 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037766 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037766 2024-03-29T06:07:20.310 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: waiting osd.3 seq 257698037766 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037766 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:20.311 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:07:20.312 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:20.707 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.c4: flush_pg_stats: test 287762808835 -lt 287762808835 2024-03-29T06:07:20.707 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:20.707 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-292057776131 2024-03-29T06:07:20.707 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-292057776131 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776131 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 292057776131' 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: waiting osd.2 seq 292057776131 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776131 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037767 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037767 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:20.708 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037767 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037767' 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: waiting osd.3 seq 257698037767 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037767 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037767 -lt 257698037767 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:20.709 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:20.713 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(containslient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:20.714 INFO:tasks.workunit.client.0.smithi123.stderr:183563: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:20.714 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:07:20.714 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:07:20.714 INFO:tasks.workunit.client.0.smithi123.stderr:183563: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:07:20.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:07:20.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:07:20.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 183562 2024-03-29T06:07:21.406 INFO:tasks.workunit.client.0.smithi123.stderr:("stale") | not)' 2024-03-29T06:07:21.406 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:21.406 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:21.406 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:07:21.406 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:21.406 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:21.406 INFO:tasks.workunit.client.0.smithi123.stderr:183565: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:21.407 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:07:21.407 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:07:21.407 INFO:tasks.workunit.client.0.smithi123.stderr:183565: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:07:21.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:07:21.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:07:21.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:07:21.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:07:21.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:07:21.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:07:21.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:07:21.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:07:21.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:21.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:21.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:21.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:22.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:22.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:07:22.355 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:07:22.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:07:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:07:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:07:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:07:22.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:07:22.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:22.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:22.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:22.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:22.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:48.630482+0000 '>' 2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:22.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:07:23.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:07:23.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:07:23.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:07:23.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:23.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:23.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:23.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:23.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:48.630482+0000 '>' 2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:23.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:07:24.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:07:24.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:07:24.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:07:24.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:24.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:24.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:24.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:25.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:48.630482+0000 '>' 2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:25.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:07:26.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:07:26.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:07:26.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:07:26.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:26.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:26.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:26.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:26.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:06:48.630482+0000 '>' 2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:26.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:07:27.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:07:27.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:07:27.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:07:27.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:27.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:27.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:27.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:27.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:07:22.533082+0000 '>' 2024-03-29T06:06:48.630482+0000 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 186436"' 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 186436' 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:07:27.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 186438"' 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 186438' 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 186436 186438' 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 186436 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/186439: /' 2024-03-29T06:07:27.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/186441: /' 2024-03-29T06:07:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:07:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: _ 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: hinfo_key 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: snapset 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:07:30.864 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:07:30.865 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:07:30.865 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:07:30.865 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:07:30.865 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:07:30.868 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:07:30.868 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:07:30.868 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:07:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:07:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:186439: start osd.1 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:07:30.913 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: _ 2024-03-29T06:07:30.914 INFO:tasks.workunit.client.0.smithi123.stderr:186441: hinfo_key 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: snapset 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:07:30.915 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:07:30.916 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:07:30.917 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:07:30.918 INFO:tasks.workunit.client.0.smithi123.stderr:186441: start osd.2 2024-03-29T06:07:35.701 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:07:35.701 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:07:35.701 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:07:35.701 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:07:35.701 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 2024-03-29T06:07:30.900+0000 7fae19a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 2024-03-29T06:07:30.900+0000 7fae19a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 2024-03-29T06:07:30.904+0000 7fae19a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 0 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.702 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 2024-03-29T06:07:31.436+0000 7fae19a5d7c0 -1 Falling back to public interface 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 1 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 2 2024-03-29T06:07:35.703 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 2024-03-29T06:07:34.396+0000 7fae19a5d7c0 -1 osd.1 70 log_to_monitors true 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 3 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.704 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186439: osd.1 up in weight 1 up_from 75 up_thru 68 down_aovery_ops 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 2024-03-29T06:07:30.940+0000 7f7ad61947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:35.706 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 2024-03-29T06:07:30.944+0000 7f7ad61947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 2024-03-29T06:07:30.948+0000 7f7ad61947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 0 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 2024-03-29T06:07:31.500+0000 7f7ad61947c0 -1 Falling back to public interface 2024-03-29T06:07:35.707 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 1 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 2 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:35.708 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 2024-03-29T06:07:34.624+0000 7f7ad61947c0 -1 osd.2 70 log_to_monitors true 2024-03-29T06:07:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:07:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 3 2024-03-29T06:07:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:07:36.756 INFO:tasks.workunit.client.0.smithi123.stderr:186441: osd.2 up in weight 1 up_from 75 up_thru 50 down_at 71 last_clean_interval [67,70) [v2:127.0.0.1:6810/1774486352,v1:127.0.0.1:6811/1774486352] [v2:127.0.0.1:6812/1774486352,v1:127.0.0.1:6813/1774486352] exists,up 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:07:36.757 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 1 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 2 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: 3' 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-03-29T06:07:36.758 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-322122547202' 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 71 last_clean_interval [68,70) [v2:127.0.0.1:6818/3397289491,v1:127.0.0.1:6819/3397289491] [v2:127.0.0.1:6820/3397289491,v1:127.0.0.1:6821/3397289491] exists,up bdf43b7d-4183-4650-a893-494f090724c4 2024-03-29T06:07:36.759 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:07:36.760 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 1 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 2 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: 3' 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:36.761 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:07:36.762 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-03-29T06:07:36.762 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-03-29T06:07:36.762 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-322122547203' 2024-03-29T06:07:36.762 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:39.189 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:07:39.189 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-03-29T06:07:39.189 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-322122547203 2-322122547202' 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037772 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037772 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-322122547203 2-322122547202 3-257698037772' 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:39.190 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: waiting osd.0 seq 21474836544 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836544 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836544 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:39.191 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547203 2024-03-29T06:07:39.192 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:39.192 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:07:39.192 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547203 2024-03-29T06:07:39.192 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:39.192 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-03-29T06:07:39.192 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2024-03-29T06:07:39.192 INFO:tasks.workunit.client.0.smithi123.stderr:186441: waiting osd.1 seq 322122547203 2024-03-29T06:07:39.193 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:07:39.193 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-03-29T06:07:39.194 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-03-29T06:07:39.194 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-03-29T06:07:39.194 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-03-29T06:07:39.194 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-322122547202 2-322122547203' 2024-03-29T06:07:39.195 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:39.195 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:07:39.195 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037773 2024-03-29T06:07:39.195 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037773 2024-03-29T06:07:39.195 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-322122547202 2-322122547203 3-257698037773' 2024-03-29T06:07:39.195 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:39.196 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-03-29T06:07:39.196 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:39.196 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:07:39.197 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-03-29T06:07:39.197 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:39.197 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-03-29T06:07:39.197 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-03-29T06:07:39.197 INFO:tasks.workunit.client.0.smithi123.stderr:186439: waiting osd.0 seq 21474836543 2024-03-29T06:07:39.197 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:39.198 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-03-29T06:07:39.198 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:07:39.199 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:07:39.199 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836543 2024-03-29T06:07:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547202 2024-03-29T06:07:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:07:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547202 2024-03-29T06:07:39.200 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-03-29T06:07:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-03-29T06:07:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:186439: waiting osd.1 seq 322122547202 2024-03-29T06:07:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:07:39.201 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-03-29T06:07:40.594 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:40.594 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547202 2024-03-29T06:07:40.594 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:40.594 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:07:40.594 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547202 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: waiting osd.2 seq 322122547202 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037772 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037772 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:40.595 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037772 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037772' 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: waiting osd.3 seq 257698037772 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037772 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:40.596 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:40.597 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:40.597 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:07:40.597 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephte$seqs 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547203 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547203 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547203' 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: waiting osd.2 seq 322122547203 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:07:40.629 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037773 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037773 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037773 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037773' 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: waiting osd.3 seq 257698037773 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037773 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:40.630 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:07:40.631 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:42.089 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:42.090 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:07:42.091 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:07:42.092 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:07:42.092 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:07:42.171 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntust/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:42.172 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:07:42.173 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:07:42.174 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:07:42.174 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:07:42.174 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:07:42.938 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:07:42.938 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:07:42.940 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:07:42.940 INFO:tasks.workunit.client.0.smithi123.stderr:186441: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:07:42.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:07:43.022 INFO:tasks.workunit.client.0.smithi123.stderr:/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:07:43.022 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:07:43.022 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:07:43.022 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:07:43.023 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:186439: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 186438 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:07:43.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:07:43.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 189359"' 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 189359' 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:43.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 189360"' 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 189360' 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 189359 189360' 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 189359 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/189362: /' 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:43.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/189364: /' 2024-03-29T06:07:46.174 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:07:46.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:07:46.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:07:46.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:07:46.179 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:07:46.179 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:07:46.180 INFO:tasks.workunit.client.0.smithi123.stderr:189364: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:07:46.181 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:07:46.182 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:07:46.182 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:07:46.182 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:07:46.182 INFO:tasks.workunit.client.0.smithi123.stderr:189364: start osd.1 2024-03-29T06:07:46.182 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:07:47.050 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:07:47.050 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:07:47.050 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:07:47.050 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:07:47.050 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:07:47.050 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:07:47.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:07:47.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:07:47.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:07:47.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:07:47.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:07:47.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:07:47.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: start osd.3 2024-03-29T06:07:47.055 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:07:51.301 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:07:51.301 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:07:51.301 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:07:51.301 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:07:51.301 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:07:51.301 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 2024-03-29T06:07:46.208+0000 7fe6c687e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 2024-03-29T06:07:46.212+0000 7fe6c687e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 2024-03-29T06:07:46.212+0000 7fe6c687e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 2024-03-29T06:07:46.748+0000 7fe6c687e7c0 -1 Falling back to public interface 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.302 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 0 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 1 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:07:51.303 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 2 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 2024-03-29T06:07:49.532+0000 7fe6c687e7c0 -1 osd.1 77 log_to_monitors true 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 3 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.304 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:07:51.955 INFO:tasks.workunit.client.0.smithi123.stderr:189364: osd.1 up in weight 1 up_from 82 up_thru 82 down_at 78 last_189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:07:51.955 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 2024-03-29T06:07:47.076+0000 7f0cb8cb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 2024-03-29T06:07:47.080+0000 7f0cb8cb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 2024-03-29T06:07:47.080+0000 7f0cb8cb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:07:51.956 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 0 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 2024-03-29T06:07:47.620+0000 7f0cb8cb27c0 -1 Falling back to public interface 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 1 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:51.957 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 2 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 2024-03-29T06:07:51.032+0000 7f0cb8cb27c0 -1 osd.3 77 log_to_monitors true 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 3 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:51.958 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:07:52.176 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [75,77) [v2:127.0.0.1:6810/4154754299,v1:127.0.0.1:6811/4154754299] [v2:127.0.0.1:6812/4154754299,v1:127.0.0.1:6813/4154754299] exists,up 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:07:52.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:07:52.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:07:52.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:07:52.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:07:52.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:07:52.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 1 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 2 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: 3' 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-29T06:07:52.178 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-352187318274' 2024-03-29T06:07:52.179 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:54.051 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph s.sh:985: wait_for_osd: sleep 1 2024-03-29T06:07:54.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:07:54.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:07:54.051 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 4 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: osd.3 up in weight 1 up_from 85 up_thru 85 down_at 78 last_clean_interval [60,77) [v2:127.0.0.1:6826/2557339502,v1:127.0.0.1:6827/2557339502] [v2:127.0.0.1:6828/2557339502,v1:127.0.0.1:6829/2557339502] exists,up 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:07:54.052 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 1 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 2 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: 3' 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:54.053 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:07:54.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-03-29T06:07:54.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-03-29T06:07:54.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-03-29T06:07:54.054 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:54.799 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-heltell osd.2 flush_pg_stats 2024-03-29T06:07:54.799 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547207 2024-03-29T06:07:54.799 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547207 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-352187318274 2-322122547207' 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-352187318274 2-322122547207 3-365072220162' 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:54.800 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: waiting osd.0 seq 21474836548 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836548 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836548 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318274 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318274 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318274' 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: waiting osd.1 seq 352187318274 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-29T06:07:54.802 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547207 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547207 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547207 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547207' 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: waiting osd.2 seq 322122547207 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547207 -lt 322122547207 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:56.175 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220162 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220162 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220162' 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: waiting osd.3 seq 365072220162 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220162 -lt 365072220162 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:56.176 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:07:56.177 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:56.190 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.cpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:07:56.190 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-03-29T06:07:56.190 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-352187318275' 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547208 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547208 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-352187318275 2-322122547208' 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-352187318275 2-322122547208 3-365072220163' 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:56.191 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: waiting osd.0 seq 21474836550 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836550 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-352187318275 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:56.192 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:07:56.193 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-352187318275 2024-03-29T06:07:56.193 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:56.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ulient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:56.538 INFO:tasks.workunit.client.0.smithi123.stderr:189364: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:56.538 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:07:56.538 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:07:56.538 INFO:tasks.workunit.client.0.smithi123.stderr:189364: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:07:56.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: waiting osd.1 seq 352187318275 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318275 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547208 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:07:57.538 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547208 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547208 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547208' 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: waiting osd.2 seq 322122547208 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547209 -lt 322122547208 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220163 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220163 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: waiting osd.3 seq 365072220163 2024-03-29T06:07:57.539 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:07:57.540 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:07:58.265 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:07:58.265 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:07:58.265 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:07:58.265 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:07:58.266 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:07:58.266 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:07:58.266 INFO:tasks.workunit.client.0.smithi123.stderr:189362: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:07:58.266 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:07:58.266 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:07:58.266 INFO:tasks.workunit.client.0.smithi123.stderr:189362: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:07:58.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:07:58.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:07:58.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 189360 2024-03-29T06:07:58.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:07:58.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:07:58.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:07:58.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:07:58.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:07:58.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:07:58.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:07:58.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:07:58.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:07:58.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:07:58.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:07:58.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:07:58.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:07:58.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:07:58.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:58.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:58.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:58.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:58.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:07:22.533082+0000 2024-03-29T06:07:58.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:07:59.210 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:07:59.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:07:22.533082+0000 2024-03-29T06:07:59.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:07:59.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:07:22.533082+0000 2024-03-29T06:07:59.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:07:59.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:07:59.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:07:59.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:07:59.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:07:59.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:07:59.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:07:59.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:07:59.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:07:22.533082+0000 '>' 2024-03-29T06:07:22.533082+0000 2024-03-29T06:07:59.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:08:00.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:08:00.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:08:00.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:08:00.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:08:00.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:08:00.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:08:00.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:08:00.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:07:22.533082+0000 '>' 2024-03-29T06:07:22.533082+0000 2024-03-29T06:08:00.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:08:01.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:08:01.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:08:01.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:08:01.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:08:01.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:08:01.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:08:01.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:08:02.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:07:22.533082+0000 '>' 2024-03-29T06:07:22.533082+0000 2024-03-29T06:08:02.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:08:03.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:08:03.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:08:03.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:08:03.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:08:03.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:08:03.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:08:03.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:08:03.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:07:22.533082+0000 '>' 2024-03-29T06:07:22.533082+0000 2024-03-29T06:08:03.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:08:04.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:08:04.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:08:04.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:08:04.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:08:04.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:08:04.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:08:04.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:08:00.068997+0000 '>' 2024-03-29T06:07:22.533082+0000 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 192235"' 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 192235' 2024-03-29T06:08:04.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 192236"' 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 192236' 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 192235 192236' 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 192235 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/192238: /' 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:08:04.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/192240: /' 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:08:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:08:07.768 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: _ 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: hinfo_key 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: snapset 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:08:07.769 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:08:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:07.772 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:08:07.773 INFO:tasks.workunit.client.0.smithi123.stderr:192238: start osd.3 2024-03-29T06:08:08.297 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:08:08.297 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:08:08.297 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:08:08.297 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:08:08.298 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: _ 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: hinfo_key 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: snapset 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:08:08.299 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:08:08.300 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:08:08.300 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:08:08.300 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:08:08.301 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:08:08.302 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:08:08.303 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:08:08.303 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:08:08.303 INFO:tasks.workunit.client.0.smithi123.stderr:192240: start osd.1 2024-03-29T06:08:12.069 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:08:12.069 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:08:12.069 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:08:12.069 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:08:12.069 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:08:12.069 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 2024-03-29T06:08:07.800+0000 7fa85f4927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 2024-03-29T06:08:07.804+0000 7fa85f4927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 2024-03-29T06:08:07.808+0000 7fa85f4927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 0 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 2024-03-29T06:08:08.391+0000 7fa85f4927c0 -1 Falling back to public interface 2024-03-29T06:08:12.070 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 1 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 2 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:08:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:12.072 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 2024-03-29T06:08:11.271+0000 7fa85f4927c0 -1 osd.3 87 log_to_monitors true 2024-03-29T06:08:12.072 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:12.072 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.072 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:08:12.072 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 3 2024-03-29T06:08:12.072 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.984 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-29T06:08:12.984 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:08:12.984 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 2024-03-29T06:08:08.331+0000 7f2aa8adb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 2024-03-29T06:08:08.335+0000 7f2aa8adb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 2024-03-29T06:08:08.335+0000 7f2aa8adb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:08:12.985 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 0 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 2024-03-29T06:08:08.883+0000 7f2aa8adb7c0 -1 Falling back to public interface 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 1 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.986 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 2 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 2024-03-29T06:08:12.091+0000 7f2aa8adb7c0 -1 osd.1 87 log_to_monitors true 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 3 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:12.987 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:08:13.754 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: osd.3 up in weight 1 up_from 92 up_thru 85 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6810/1444562830,v1:127.0.0.1:6811/1444562830] [v2:127.0.0.1:6812/1444562830,v1:127.0.0.1:6813/1444562830] exists,up 6587b6c5-5bfe-48b7-8cc9-cb634c382244 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:08:13.755 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 1 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 2 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: 3' 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836554 2024-03-29T06:08:13.756 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836554 2024-03-29T06:08:13.757 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554' 2024-03-29T06:08:13.757 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:13.757 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:08:13.757 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=2 2024-03-29T06:08:13.757 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 2 2024-03-29T06:08:13.757 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-2' 2024-03-29T06:08:13.757 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cepheph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 4 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: osd.1 up in weight 1 up_from 95 up_thru 82 down_at 88 last_clean_interval [82,87) [v2:127.0.0.1:6826/583065461,v1:127.0.0.1:6827/583065461] [v2:127.0.0.1:6828/583065461,v1:127.0.0.1:6829/583065461] exists,up 9d702115-ae14-4838-8542-30bdcb55a5a6 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:08:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:08:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 1 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 2 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: 3' 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:15.202 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:08:15.203 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-03-29T06:08:15.203 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-03-29T06:08:15.203 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-03-29T06:08:15.203 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:16.469 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:08:16.469 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-03-29T06:08:16.469 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-03-29T06:08:16.469 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-408021893123' 2024-03-29T06:08:16.469 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:16.469 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:08:16.469 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547215 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547215 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-408021893123 2-322122547215' 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991235 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991235 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-408021893123 2-322122547215 3-395136991235' 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:16.470 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: waiting osd.0 seq 21474836556 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-408021893123 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-408021893123 2024-03-29T06:08:16.471 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:16.472 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-03-29T06:08:16.472 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 408021893123' 2024-03-29T06:08:16.472 INFO:tasks.workunit.client.0.smithi123.stderr:192240: waiting osd.1 seq 408021893123 2024-03-29T06:08:16.472 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:08:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.test/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:08:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547213 2024-03-29T06:08:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547213 2024-03-29T06:08:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-2 2-322122547213' 2024-03-29T06:08:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:08:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:08:16.695 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-2 2-322122547213 3-395136991234' 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836554 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836554 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836554 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: waiting osd.0 seq 21474836554 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836554 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:16.696 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-2 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-2 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=2 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: waiting osd.1 seq 2 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 2 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 2 2024-03-29T06:08:16.697 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:18.030 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547213 2024-03-29T06:08:18.030 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547213 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547213 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547213' 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: waiting osd.2 seq 322122547213 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547215 -lt 322122547213 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991234 2024-03-29T06:08:18.031 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991234 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991234' 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: waiting osd.3 seq 395136991234 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:08:18.032 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:08:18.033 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:08:18.898 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helsh:2214: flush_pg_stats: test 408021893123 -lt 408021893123 2024-03-29T06:08:18.898 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:18.898 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547215 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547215 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547215 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547215' 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: waiting osd.2 seq 322122547215 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547215 -lt 322122547215 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991235 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:08:18.899 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991235 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991235 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: waiting osd.3 seq 395136991235 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991235 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991236 -lt 395136991235 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:08:18.900 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:08:18.901 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:08:18.901 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:08:18.901 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:08:19.638 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(copers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:08:19.638 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:08:19.638 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:08:19.639 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-29T06:08:19.640 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hentains("stale") | not)' 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:08:20.291 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:08:20.292 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:08:20.293 INFO:tasks.workunit.client.0.smithi123.stderr:192240: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:08:20.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:08:20.430 INFO:tasks.workunit.client.0.smithi123.stderr:lpers.sh:1663: wait_for_clean: loop=0 2024-03-29T06:08:20.430 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:08:20.430 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:08:20.430 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:08:20.431 INFO:tasks.workunit.client.0.smithi123.stderr:192238: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 192236 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:08:20.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:08:20.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:08:20.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:08:20.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:08:20.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:08:20.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:08:20.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:08:20.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:08:20.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:08:20.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:08:20.809 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:08:20.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:08:20.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:08:20.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:08:20.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:08:20.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:08:20.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:08:20.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:08:20.851 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:08:20.851 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:20.851 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:20.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:08:20.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:08:20.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:08:20.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:08:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:08:20.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:08:20.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:08:20.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:08:20.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:08:20.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:08:20.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:08:20.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:08:20.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:08:20.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:08:20.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:08:20.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:08:20.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:08:20.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:08:20.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:08:20.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:08:20.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:20.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:20.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:08:20.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:08:20.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:08:20.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:08:20.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:08:20.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:20.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:20.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:08:20.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:08:20.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:08:20.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T06:08:21.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:08:21.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:08:21.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:08:21.243 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:08:21.243 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:21.243 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:21.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:21.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T06:08:21.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:08:21.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:08:21.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:08:21.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:08:21.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:21.489 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:08:21.490 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:08:21.490 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:08:21.602 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:08:21.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:08:21.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:08:21.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:08:21.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:08:21.604 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:08:21.604 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:08:21.604 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:08:21.604 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:08:21.604 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:21.604 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:21.605 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:08:21.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:08:21.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:08:21.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-29T06:08:21.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:08:21.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:08:21.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:08:21.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:08:21.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:08:21.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:08:22.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:08:22.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:08:22.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:08:22.672 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:08:22.672 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:22.672 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:22.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:22.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:08:22.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:08:22.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:08:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:08:22.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:22.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:08:22.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:08:22.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:08:22.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:08:22.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:08:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:08:22.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:08:22.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef0f50bf-5f0e-4cb3-97f2-4fdb7850e90f 2024-03-29T06:08:22.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ef0f50bf-5f0e-4cb3-97f2-4fdb7850e90f' 2024-03-29T06:08:22.717 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 ef0f50bf-5f0e-4cb3-97f2-4fdb7850e90f 2024-03-29T06:08:22.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:08:22.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWWgZmAPSkKxAAWGnW2OYAJfE0RRGs9SWYYw== 2024-03-29T06:08:22.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWWgZmAPSkKxAAWGnW2OYAJfE0RRGs9SWYYw=="}' 2024-03-29T06:08:22.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef0f50bf-5f0e-4cb3-97f2-4fdb7850e90f -i td/osd-scrub-repair/0/new.json 2024-03-29T06:08:23.048 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:08:23.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:08:23.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWWgZmAPSkKxAAWGnW2OYAJfE0RRGs9SWYYw== --osd-uuid ef0f50bf-5f0e-4cb3-97f2-4fdb7850e90f 2024-03-29T06:08:23.099 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:23.095+0000 7f41631df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:23.141 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:23.135+0000 7f41631df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:23.142 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:23.139+0000 7f41631df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:23.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:23.139+0000 7f41631df7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:08:23.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:23.139+0000 7f41631df7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:08:23.143 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:23.139+0000 7f41631df7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:08:23.257 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:23.251+0000 7f41631df7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:08:30.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:08:30.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:08:30.999 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:08:31.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:08:31.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:08:31.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:08:31.376 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:08:31.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:08:31.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:08:31.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:08:31.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:08:31.407 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:31.403+0000 7f3cbbb077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:31.410 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:31.407+0000 7f3cbbb077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:31.411 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:31.407+0000 7f3cbbb077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:08:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:08:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:08:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:08:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:08:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:08:31.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:31.684 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:08:31.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:08:31.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:31.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:31.944 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:31.939+0000 7f3cbbb077c0 -1 Falling back to public interface 2024-03-29T06:08:31.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:32.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:32.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:32.980 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:08:32.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:08:32.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:32.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:33.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:34.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:34.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:34.287 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:08:34.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:08:34.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:34.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:34.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:34.843 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:08:34.839+0000 7f3cbbb077c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:08:35.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:35.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:35.605 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:08:35.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:08:35.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:35.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:36.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:37.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:37.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:37.024 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:08:37.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:08:37.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:37.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:37.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:38.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:38.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:38.342 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:08:38.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:08:38.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:38.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:38.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:39.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:39.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:39.699 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:08:39.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:08:39.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:39.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:40.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:41.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:41.168 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:08:41.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:08:41.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:41.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:41.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:42.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:42.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:42.487 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:08:42.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:08:42.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:42.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:42.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:43.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:43.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:43.808 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:08:43.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:08:43.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:43.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:44.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:45.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:45.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:45.174 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:08:45.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:08:45.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:45.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:45.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:46.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:46.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:46.497 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:08:46.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:08:46.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:46.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:47.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:48.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:48.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:48.010 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:08:48.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:08:48.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:48.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:48.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:49.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:49.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:49.603 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:08:49.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:08:49.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:49.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:50.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:51.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:51.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:51.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:08:51.018 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:08:51.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:51.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:51.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:52.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:52.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:52.366 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:08:52.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:08:52.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:52.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:52.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:53.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:53.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:53.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:08:53.719 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:08:53.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:53.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:54.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:55.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:55.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:55.337 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:08:55.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:08:55.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:55.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:55.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:56.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:56.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:56.936 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:08:56.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:08:56.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:56.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:57.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:58.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:58.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:58.314 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:08:58.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:08:58.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:58.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:08:58.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:08:59.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:08:59.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:08:59.948 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:08:59.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:08:59.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:08:59.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:09:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:01.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:01.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:01.459 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:09:01.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:09:01.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:01.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:09:02.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:03.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:03.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:03.110 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:09:03.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:09:03.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:03.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:09:03.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:04.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:04.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:04.819 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:09:04.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:09:04.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:04.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:09:05.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:06.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:06.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:06.648 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:09:06.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:09:06.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:06.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:09:06.970 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3420700258,v1:127.0.0.1:6803/3420700258] [v2:127.0.0.1:6804/3420700258,v1:127.0.0.1:6805/3420700258] exists,up ef0f50bf-5f0e-4cb3-97f2-4fdb7850e90f 2024-03-29T06:09:06.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:09:06.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:09:06.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:09:06.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:09:06.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:09:06.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:09:06.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25e23f7d-fbc1-4ecd-870d-f3ca94b77436 2024-03-29T06:09:06.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 25e23f7d-fbc1-4ecd-870d-f3ca94b77436' 2024-03-29T06:09:06.977 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 25e23f7d-fbc1-4ecd-870d-f3ca94b77436 2024-03-29T06:09:06.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:09:06.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACWwZmKzk1OxAAViVjQG/1yTbx1cjNolF0rA== 2024-03-29T06:09:06.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACWwZmKzk1OxAAViVjQG/1yTbx1cjNolF0rA=="}' 2024-03-29T06:09:06.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25e23f7d-fbc1-4ecd-870d-f3ca94b77436 -i td/osd-scrub-repair/1/new.json 2024-03-29T06:09:07.611 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:09:07.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:09:07.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACWwZmKzk1OxAAViVjQG/1yTbx1cjNolF0rA== --osd-uuid 25e23f7d-fbc1-4ecd-870d-f3ca94b77436 2024-03-29T06:09:07.671 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:07.670+0000 7f428304d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:07.785 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:07.782+0000 7f428304d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:07.786+0000 7f428304d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:07.786+0000 7f428304d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:09:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:07.786+0000 7f428304d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:09:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:07.786+0000 7f428304d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:09:07.885 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:07.886+0000 7f428304d7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:09:15.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:09:15.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:09:15.914 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:09:15.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:09:15.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:09:16.285 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:09:16.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:09:16.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:09:16.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:09:16.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:09:16.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:09:16.319 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:16.318+0000 7fdcafb707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:16.376 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:16.374+0000 7fdcafb707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:16.378 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:16.378+0000 7fdcafb707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:09:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:09:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:09:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:09:16.579 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:09:16.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:09:16.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:16.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:09:16.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:16.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:16.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:16.948 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:16.946+0000 7fdcafb707c0 -1 Falling back to public interface 2024-03-29T06:09:17.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:17.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:17.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:09:17.877 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:09:17.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:17.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:18.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:19.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:19.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:19.185 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:09:19.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:09:19.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:19.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:19.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:20.172 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:20.170+0000 7fdcafb707c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:09:20.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:20.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:20.678 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:09:20.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:09:20.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:20.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:22.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:22.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:22.212 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:09:22.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:09:22.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:22.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:22.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:23.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:23.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:23.518 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:09:23.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:09:23.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:23.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:23.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:24.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:24.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:24.876 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:09:24.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:09:24.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:24.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:25.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:26.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:26.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:26.343 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:09:26.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:09:26.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:26.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:26.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:27.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:27.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:27.676 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:09:27.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:09:27.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:27.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:28.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:29.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:29.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:29.134 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:09:29.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:09:29.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:29.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:29.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:30.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:30.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:30.567 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:09:30.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:09:30.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:30.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:30.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:31.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:31.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:31.891 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:09:31.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:09:31.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:31.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:32.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:33.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:33.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:33.352 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:09:33.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:09:33.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:33.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:33.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:34.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:34.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:34.672 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:09:34.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:09:34.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:34.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:34.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:35.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:35.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:35.982 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:09:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:09:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:35.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:36.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:37.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:37.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:37.290 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:09:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:09:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:37.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:38.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:38.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:38.610 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:09:38.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:09:38.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:38.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:38.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:39.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:39.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:39.930 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:09:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:09:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:40.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:41.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:41.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:41.296 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:09:41.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:09:41.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:41.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:42.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:43.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:43.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:43.267 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:09:43.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:09:43.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:43.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:43.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:44.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:44.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:44.657 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:09:44.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:09:44.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:44.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:45.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:46.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:46.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:46.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:09:46.190 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:09:46.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:46.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:47.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:48.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:48.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:48.015 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:09:48.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:09:48.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:48.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:48.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:49.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:49.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:49.696 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:09:49.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:09:49.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:49.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:50.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:51.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:51.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:51.149 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:09:51.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:09:51.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:51.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:51.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:09:52.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:09:52.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:09:52.811 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:09:52.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:09:52.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:09:52.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:09:53.128 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2454542331,v1:127.0.0.1:6811/2454542331] [v2:127.0.0.1:6812/2454542331,v1:127.0.0.1:6813/2454542331] exists,up 25e23f7d-fbc1-4ecd-870d-f3ca94b77436 2024-03-29T06:09:53.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:09:53.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:09:53.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:09:53.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:09:53.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:09:53.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:09:53.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:09:53.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:09:53.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:09:53.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:09:53.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:09:53.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9c3273e-cd32-4d83-aaee-35f3d0bf0ddf 2024-03-29T06:09:53.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e9c3273e-cd32-4d83-aaee-35f3d0bf0ddf' 2024-03-29T06:09:53.135 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 e9c3273e-cd32-4d83-aaee-35f3d0bf0ddf 2024-03-29T06:09:53.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:09:53.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxWwZmXqIBCRAA5Q8NQlvQPV7IqBM5tFCTzQ== 2024-03-29T06:09:53.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxWwZmXqIBCRAA5Q8NQlvQPV7IqBM5tFCTzQ=="}' 2024-03-29T06:09:53.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9c3273e-cd32-4d83-aaee-35f3d0bf0ddf -i td/osd-scrub-repair/2/new.json 2024-03-29T06:09:53.559 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:09:53.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T06:09:53.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxWwZmXqIBCRAA5Q8NQlvQPV7IqBM5tFCTzQ== --osd-uuid e9c3273e-cd32-4d83-aaee-35f3d0bf0ddf 2024-03-29T06:09:53.614 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:53.613+0000 7fe456ffe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:53.686 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:53.685+0000 7fe456ffe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:53.688 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:53.685+0000 7fe456ffe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:09:53.688 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:53.685+0000 7fe456ffe7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:09:53.688 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:53.685+0000 7fe456ffe7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:09:53.688 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:53.685+0000 7fe456ffe7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:09:53.772 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:09:53.769+0000 7fe456ffe7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T06:10:02.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T06:10:02.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:10:02.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T06:10:02.289 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T06:10:02.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:10:02.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T06:10:02.682 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T06:10:02.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:10:02.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:10:02.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:10:02.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:10:02.714 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:02.713+0000 7fcbc3bd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:02.758 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:02.757+0000 7fcbc3bd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:02.759 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:02.757+0000 7fcbc3bd17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:02.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T06:10:02.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:10:02.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:10:02.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:10:02.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:10:02.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:02.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:10:02.992 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:10:02.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:02.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:03.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:03.346 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:03.345+0000 7fcbc3bd17c0 -1 Falling back to public interface 2024-03-29T06:10:04.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:04.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:04.297 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:10:04.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:10:04.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:04.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:04.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:05.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:05.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:05.598 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:10:05.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:10:05.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:05.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:06.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:06.380 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:06.377+0000 7fcbc3bd17c0 -1 osd.2 0 log_to_monitors true 2024-03-29T06:10:07.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:07.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:07.042 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:10:07.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:10:07.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:07.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:07.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:08.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:08.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:08.443 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:10:08.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:10:08.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:08.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:08.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:09.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:09.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:09.777 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:10:09.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:10:09.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:09.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:10.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:11.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:11.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:11.298 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:10:11.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:10:11.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:11.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:11.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:12.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:12.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:12.932 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:10:12.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:10:12.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:12.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:13.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:14.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:14.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:14.267 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:10:14.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:10:14.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:14.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:14.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:15.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:15.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:15.623 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:10:15.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:10:15.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:15.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:15.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:16.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:16.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:16.938 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:10:16.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:10:16.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:16.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:17.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:18.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:18.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:18.288 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:10:18.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:10:18.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:18.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:18.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:19.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:19.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:19.752 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:10:19.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:10:19.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:19.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:20.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:21.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:21.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:21.260 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:10:21.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:10:21.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:21.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:21.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:22.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:22.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:22.990 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:10:22.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:10:22.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:22.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:23.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:24.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:24.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:24.319 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:10:24.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:10:24.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:24.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:24.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:25.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:25.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:25.688 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:10:25.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:10:25.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:25.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:26.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:27.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:27.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:27.723 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:10:27.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:10:27.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:27.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:28.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:29.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:29.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:29.476 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:10:29.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:10:29.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:29.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:30.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:31.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:31.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:31.118 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:10:31.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:10:31.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:31.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:31.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:32.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:32.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:32.811 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:10:32.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:10:32.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:32.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:33.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:34.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:34.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:34.530 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:10:34.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:10:34.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:34.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:34.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:35.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:35.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:35.961 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:10:35.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:10:35.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:35.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:36.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:37.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:37.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:37.796 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:10:37.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:10:37.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:37.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:38.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:39.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:39.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:39.165 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:10:39.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:10:39.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:39.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:10:39.578 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1215614505,v1:127.0.0.1:6819/1215614505] [v2:127.0.0.1:6820/1215614505,v1:127.0.0.1:6821/1215614505] exists,up e9c3273e-cd32-4d83-aaee-35f3d0bf0ddf 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:10:39.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:10:39.583 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:10:39.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:10:39.585 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:10:39.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75727be0-e71a-4f24-99d4-cdaee9b3727e 2024-03-29T06:10:39.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 75727be0-e71a-4f24-99d4-cdaee9b3727e' 2024-03-29T06:10:39.585 INFO:tasks.workunit.client.0.smithi123.stdout:add osd3 75727be0-e71a-4f24-99d4-cdaee9b3727e 2024-03-29T06:10:39.586 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:10:39.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfWwZm3qYgJBAAnQJ8vpd0Y6T6cJ3oR2iHFA== 2024-03-29T06:10:39.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfWwZm3qYgJBAAnQJ8vpd0Y6T6cJ3oR2iHFA=="}' 2024-03-29T06:10:39.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75727be0-e71a-4f24-99d4-cdaee9b3727e -i td/osd-scrub-repair/3/new.json 2024-03-29T06:10:40.052 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:10:40.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-29T06:10:40.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfWwZm3qYgJBAAnQJ8vpd0Y6T6cJ3oR2iHFA== --osd-uuid 75727be0-e71a-4f24-99d4-cdaee9b3727e 2024-03-29T06:10:40.109 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:40.104+0000 7ff824b577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:40.188 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:40.184+0000 7ff824b577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:40.190 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:40.188+0000 7ff824b577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:40.191 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:40.188+0000 7ff824b577c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:10:40.191 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:40.188+0000 7ff824b577c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:10:40.191 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:40.188+0000 7ff824b577c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:10:40.286 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:40.280+0000 7ff824b577c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-29T06:10:48.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-29T06:10:48.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:10:48.633 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd3 key to auth repository 2024-03-29T06:10:48.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-29T06:10:48.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:10:49.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-29T06:10:49.022 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.3 2024-03-29T06:10:49.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:10:49.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:10:49.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:10:49.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:10:49.055 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:49.052+0000 7fb84db6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:49.057 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:49.056+0000 7fb84db6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:49.058 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:49.056+0000 7fb84db6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:10:49.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-29T06:10:49.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:10:49.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:10:49.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:10:49.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:10:49.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:49.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:10:49.319 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:10:49.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:49.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:49.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:49.639 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:49.636+0000 7fb84db6d7c0 -1 Falling back to public interface 2024-03-29T06:10:50.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:50.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:50.611 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:10:50.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:10:50.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:50.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:50.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:51.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:51.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:51.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:10:51.910 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:10:51.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:51.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:52.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:52.630 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:10:52.628+0000 7fb84db6d7c0 -1 osd.3 0 log_to_monitors true 2024-03-29T06:10:53.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:53.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:53.242 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:10:53.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:10:53.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:53.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:54.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:54.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:54.757 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:10:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:10:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:55.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:56.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:56.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:56.181 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:10:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:10:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:56.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:57.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:57.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:57.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:10:57.725 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:10:57.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:57.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:58.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:10:59.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:10:59.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:10:59.146 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:10:59.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:10:59.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:10:59.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:10:59.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:00.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:00.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:00.459 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:11:00.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:11:00.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:00.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:01.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:01.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:01.842 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:11:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:11:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:02.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:03.146 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:11:03.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:11:03.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:03.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:03.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:04.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:04.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:04.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:11:04.450 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:11:04.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:04.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:04.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:05.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:05.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:05.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:11:05.839 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:11:05.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:05.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:06.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:07.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:07.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:07.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:11:07.365 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:11:07.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:07.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:07.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:08.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:08.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:08.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:11:08.681 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:11:08.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:08.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:09.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:10.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:10.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:10.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:11:10.051 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:11:10.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:10.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:10.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:11.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:11.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:11.498 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:11:11.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:11:11.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:11.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:11.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:12.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:12.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:12.917 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:11:12.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:11:12.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:12.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:13.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:14.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:14.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:14.238 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:11:14.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:11:14.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:14.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:14.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:15.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:15.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:15.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:11:15.762 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:11:15.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:15.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:16.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:17.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:17.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:17.797 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:11:17.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:11:17.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:17.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:18.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:19.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:19.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:19.400 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:11:19.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:11:19.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:19.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:20.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:11:21.274 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:11:21.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:21.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:22.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:23.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:23.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:23.349 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:11:23.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:11:23.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:23.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:23.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:24.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:24.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:24.651 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:11:24.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:11:24.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:24.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:25.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:26.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:26.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:26.303 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:11:26.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:11:26.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:26.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:11:26.658 INFO:tasks.workunit.client.0.smithi123.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4263002857,v1:127.0.0.1:6827/4263002857] [v2:127.0.0.1:6828/4263002857,v1:127.0.0.1:6829/4263002857] exists,up 75727be0-e71a-4f24-99d4-cdaee9b3727e 2024-03-29T06:11:26.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:11:26.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:11:26.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-29T06:11:26.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:11:26.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:11:26.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-29T06:11:26.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:11:26.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7d41da0-b713-4a17-912d-768f54bf3298 2024-03-29T06:11:26.666 INFO:tasks.workunit.client.0.smithi123.stdout:add osd4 e7d41da0-b713-4a17-912d-768f54bf3298 2024-03-29T06:11:26.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 e7d41da0-b713-4a17-912d-768f54bf3298' 2024-03-29T06:11:26.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:11:26.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOWwZmxu35KBAAnAyt6+bU1nq1t0MV81pQZg== 2024-03-29T06:11:26.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOWwZmxu35KBAAnAyt6+bU1nq1t0MV81pQZg=="}' 2024-03-29T06:11:26.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7d41da0-b713-4a17-912d-768f54bf3298 -i td/osd-scrub-repair/4/new.json 2024-03-29T06:11:27.161 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:11:27.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-29T06:11:27.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOWwZmxu35KBAAnAyt6+bU1nq1t0MV81pQZg== --osd-uuid e7d41da0-b713-4a17-912d-768f54bf3298 2024-03-29T06:11:27.221 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:27.219+0000 7fb7f0f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:11:27.321 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:27.319+0000 7fb7f0f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:11:27.323 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:27.319+0000 7fb7f0f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:11:27.323 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:27.319+0000 7fb7f0f1f7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-29T06:11:27.323 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:27.319+0000 7fb7f0f1f7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-29T06:11:27.323 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:27.319+0000 7fb7f0f1f7c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-29T06:11:27.409 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:27.407+0000 7fb7f0f1f7c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-29T06:11:35.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-29T06:11:35.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:11:35.617 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd4 key to auth repository 2024-03-29T06:11:35.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-29T06:11:35.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:11:36.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-29T06:11:36.000 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.4 2024-03-29T06:11:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:11:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:11:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:11:36.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:11:36.032 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:36.027+0000 7fcf0ffb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:11:36.034 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:36.031+0000 7fcf0ffb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:11:36.036 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:36.031+0000 7fcf0ffb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:11:36.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-29T06:11:36.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:11:36.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-29T06:11:36.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:11:36.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:11:36.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:36.312 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:11:36.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:11:36.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:36.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:36.603 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:36.599+0000 7fcf0ffb27c0 -1 Falling back to public interface 2024-03-29T06:11:36.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:37.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:37.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:37.680 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:11:37.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:11:37.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:37.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:38.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:39.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:39.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:39.045 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:11:39.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:11:39.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:39.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:39.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:39.590 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:11:39.587+0000 7fcf0ffb27c0 -1 osd.4 0 log_to_monitors true 2024-03-29T06:11:40.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:40.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:40.392 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:11:40.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:11:40.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:40.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:40.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:41.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:41.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:41.798 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:11:41.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:11:41.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:41.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:42.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:43.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:43.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:43.119 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:11:43.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:11:43.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:43.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:43.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:44.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:44.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:44.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:11:44.437 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:11:44.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:44.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:44.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:45.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:45.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:45.943 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:11:45.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:11:45.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:45.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:46.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:47.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:47.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:47.439 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:11:47.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:11:47.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:47.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:47.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:48.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:48.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:48.957 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:11:48.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:11:48.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:48.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:49.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:50.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:50.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:50.279 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:11:50.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:11:50.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:50.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:50.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:51.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:51.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:51.683 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:11:51.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:11:51.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:51.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:52.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:53.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:53.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:53.144 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:11:53.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:11:53.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:53.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:53.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:54.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:54.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:54.558 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:11:54.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:11:54.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:54.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:54.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:55.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:55.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:55.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:11:55.904 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:11:55.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:55.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:56.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:57.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:57.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:57.428 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:11:57.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:11:57.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:57.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:57.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:11:58.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:11:58.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:11:58.743 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:11:58.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:11:58.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:11:58.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:11:59.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:00.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:00.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:00.136 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:12:00.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:12:00.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:00.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:01.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:12:01.843 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:12:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:02.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:03.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:03.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:03.957 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:12:03.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:12:03.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:03.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:06.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:06.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:06.040 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:12:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:12:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:07.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:08.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:08.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:08.138 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:12:08.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:12:08.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:08.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:08.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:09.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:09.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:09.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:12:09.879 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:12:09.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:09.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:10.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:11.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:11.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:11.700 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:12:11.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:12:11.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:11.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:12.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:13.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:13.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:13.480 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:12:13.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:12:13.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:13.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:12:13.932 INFO:tasks.workunit.client.0.smithi123.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4200790036,v1:127.0.0.1:6835/4200790036] [v2:127.0.0.1:6836/4200790036,v1:127.0.0.1:6837/4200790036] exists,up e7d41da0-b713-4a17-912d-768f54bf3298 2024-03-29T06:12:13.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:12:13.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:12:13.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:12:13.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:12:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:12:13.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:12:13.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:12:13.940 INFO:tasks.workunit.client.0.smithi123.stdout:add osd5 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:12:13.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 1c46e833-5128-4294-b759-7c4a6d7535b2' 2024-03-29T06:12:13.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:12:13.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9WwZmbeJHORAAPRl4cpD5VmSi6l20Lw5+Cg== 2024-03-29T06:12:13.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9WwZmbeJHORAAPRl4cpD5VmSi6l20Lw5+Cg=="}' 2024-03-29T06:12:13.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c46e833-5128-4294-b759-7c4a6d7535b2 -i td/osd-scrub-repair/5/new.json 2024-03-29T06:12:14.384 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:12:14.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-29T06:12:14.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9WwZmbeJHORAAPRl4cpD5VmSi6l20Lw5+Cg== --osd-uuid 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:12:14.440 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:14.438+0000 7fbe1dfff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:14.519 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:14.518+0000 7fbe1dfff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:14.520 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:14.518+0000 7fbe1dfff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:14.521 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:14.522+0000 7fbe1dfff7c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-29T06:12:14.521 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:14.522+0000 7fbe1dfff7c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-29T06:12:14.521 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:14.522+0000 7fbe1dfff7c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-29T06:12:14.626 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:14.626+0000 7fbe1dfff7c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-29T06:12:22.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-29T06:12:22.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:12:22.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-29T06:12:22.341 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd5 key to auth repository 2024-03-29T06:12:22.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:12:22.733 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.5 2024-03-29T06:12:22.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-29T06:12:22.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:12:22.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:12:22.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:12:22.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:12:22.766 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:22.766+0000 7f9676fd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:22.768 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:22.766+0000 7f9676fd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:22.769 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:22.770+0000 7f9676fd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:23.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-29T06:12:23.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:12:23.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:12:23.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:12:23.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:12:23.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:23.046 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:12:23.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:12:23.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:23.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:23.328 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:23.326+0000 7f9676fd87c0 -1 Falling back to public interface 2024-03-29T06:12:23.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:24.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:24.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:24.354 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:12:24.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:12:24.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:24.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:24.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:25.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:25.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:25.673 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:12:25.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:12:25.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:25.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:26.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:26.090+0000 7f9676fd87c0 -1 osd.5 0 log_to_monitors true 2024-03-29T06:12:27.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:27.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:27.090 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:12:27.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:12:27.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:27.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:27.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:28.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:28.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:28.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:12:28.484 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:12:28.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:28.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:28.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:29.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:29.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:29.808 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:12:29.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:12:29.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:29.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:30.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:31.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:31.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:31.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:12:31.126 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:12:31.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:31.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:31.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:32.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:32.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:32.449 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:12:32.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:12:32.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:32.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:32.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:33.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:33.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:33.771 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:12:33.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:12:33.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:33.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:34.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:35.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:35.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:35.168 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:12:35.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:12:35.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:35.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:35.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:36.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:36.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:36.475 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:12:36.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:12:36.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:36.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:36.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:37.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:37.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:37.795 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:12:37.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:12:37.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:37.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:38.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:39.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:39.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:39.157 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:12:39.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:12:39.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:39.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:39.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:40.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:40.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:40.477 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:12:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:12:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:40.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:41.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:41.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:41.804 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:12:41.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:12:41.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:41.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:42.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:43.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:43.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:43.563 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:12:43.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:12:43.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:43.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:44.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:45.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:45.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:45.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:12:45.355 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:12:45.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:45.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:45.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:46.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:46.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:46.694 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:12:46.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:12:46.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:46.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:47.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:48.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:48.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:48.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:12:48.474 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:12:48.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:48.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:49.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:50.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:50.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:50.222 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:12:50.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:12:50.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:50.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:50.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:51.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:51.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:51.971 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:12:51.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:12:51.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:51.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:52.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:12:53.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:12:53.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:12:53.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:12:53.426 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:12:53.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:12:53.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:12:53.744 INFO:tasks.workunit.client.0.smithi123.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/969587156,v1:127.0.0.1:6843/969587156] [v2:127.0.0.1:6844/969587156,v1:127.0.0.1:6845/969587156] exists,up 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:12:53.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:12:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:12:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:12:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:12:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-29T06:12:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-29T06:12:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:12:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:12:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:12:53.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:12:53.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:12:53.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:12:53.751 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:12:53.751 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:12:53.751 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:12:53.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:12:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-29T06:12:53.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:12:53.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e8eda09b-8a8f-484f-9da6-f8643bd6d1be 2024-03-29T06:12:53.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 e8eda09b-8a8f-484f-9da6-f8643bd6d1be' 2024-03-29T06:12:53.753 INFO:tasks.workunit.client.0.smithi123.stdout:add osd6 e8eda09b-8a8f-484f-9da6-f8643bd6d1be 2024-03-29T06:12:53.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:12:53.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlWwZmZnnzLRAADwb7im6J2cS+fhXQ4ovYWA== 2024-03-29T06:12:53.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlWwZmZnnzLRAADwb7im6J2cS+fhXQ4ovYWA=="}' 2024-03-29T06:12:53.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e8eda09b-8a8f-484f-9da6-f8643bd6d1be -i td/osd-scrub-repair/6/new.json 2024-03-29T06:12:54.367 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:12:54.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-29T06:12:54.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlWwZmZnnzLRAADwb7im6J2cS+fhXQ4ovYWA== --osd-uuid e8eda09b-8a8f-484f-9da6-f8643bd6d1be 2024-03-29T06:12:54.423 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:54.421+0000 7f9172f027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:54.511 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:54.509+0000 7f9172f027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:54.513 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:54.509+0000 7f9172f027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:12:54.513 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:54.513+0000 7f9172f027c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-29T06:12:54.513 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:54.513+0000 7f9172f027c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-29T06:12:54.513 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:54.513+0000 7f9172f027c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-29T06:12:54.612 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:12:54.609+0000 7f9172f027c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-29T06:13:00.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-29T06:13:00.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:13:00.983 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd6 key to auth repository 2024-03-29T06:13:00.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-29T06:13:00.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:13:01.375 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.6 2024-03-29T06:13:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-29T06:13:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:13:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:13:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:13:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:13:01.407 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:01.405+0000 7fc297e2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:01.451 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:01.449+0000 7fc297e2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:01.453 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:01.453+0000 7fc297e2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:01.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-29T06:13:01.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:13:01.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-29T06:13:01.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:13:01.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:13:01.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:01.678 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:13:01.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:13:01.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:01.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:01.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:02.023 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:02.021+0000 7fc297e2c7c0 -1 Falling back to public interface 2024-03-29T06:13:02.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:02.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:02.977 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:13:02.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:13:02.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:02.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:03.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:04.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:04.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:04.295 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:13:04.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:13:04.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:04.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:04.506 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:04.505+0000 7fc297e2c7c0 -1 osd.6 0 log_to_monitors true 2024-03-29T06:13:04.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:05.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:05.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:05.626 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:13:05.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:13:05.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:05.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:06.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:07.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:07.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:07.128 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:13:07.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:13:07.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:07.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:07.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:08.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:08.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:08.629 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:13:08.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:13:08.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:08.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:09.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:09.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:09.950 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:13:09.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:13:09.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:09.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:10.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:11.426 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:13:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:13:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:11.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:12.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:12.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:12.827 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:13:12.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:13:12.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:12.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:13.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:14.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:14.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:14.145 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:13:14.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:13:14.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:14.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:14.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:15.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:15.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:15.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:13:15.611 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:13:15.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:15.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:16.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:17.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:17.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:17.037 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:13:17.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:13:17.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:17.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:17.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:18.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:18.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:18.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:13:18.350 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:13:18.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:18.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:18.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:19.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:19.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:19.690 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:13:19.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:13:19.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:19.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:20.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:21.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:21.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:21.120 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:13:21.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:13:21.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:21.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:22.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:23.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:23.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:23.034 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:13:23.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:13:23.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:23.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:23.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:24.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:24.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:24.804 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:13:24.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:13:24.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:24.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:25.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:26.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:26.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:26.600 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:13:26.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:13:26.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:26.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:27.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:28.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:28.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:28.021 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:13:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:13:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:28.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:28.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:29.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:29.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:29.927 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:13:29.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:13:29.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:29.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:30.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:31.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:31.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:31.656 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:13:31.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:13:31.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:31.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:32.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:33.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:33.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:33.500 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:13:33.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:13:33.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:33.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:13:33.952 INFO:tasks.workunit.client.0.smithi123.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2647548738,v1:127.0.0.1:6851/2647548738] [v2:127.0.0.1:6852/2647548738,v1:127.0.0.1:6853/2647548738] exists,up e8eda09b-8a8f-484f-9da6-f8643bd6d1be 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:13:33.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:13:33.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:13:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:13:33.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-29T06:13:33.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:13:33.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=efbfff5e-690e-487a-8e84-394ffc11f9aa 2024-03-29T06:13:33.959 INFO:tasks.workunit.client.0.smithi123.stdout:add osd7 efbfff5e-690e-487a-8e84-394ffc11f9aa 2024-03-29T06:13:33.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 efbfff5e-690e-487a-8e84-394ffc11f9aa' 2024-03-29T06:13:33.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:13:33.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANXAZmMGVrOhAAIwcpo01IlN/NIVdE0a+rGw== 2024-03-29T06:13:33.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANXAZmMGVrOhAAIwcpo01IlN/NIVdE0a+rGw=="}' 2024-03-29T06:13:33.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new efbfff5e-690e-487a-8e84-394ffc11f9aa -i td/osd-scrub-repair/7/new.json 2024-03-29T06:13:34.399 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:13:34.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-29T06:13:34.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANXAZmMGVrOhAAIwcpo01IlN/NIVdE0a+rGw== --osd-uuid efbfff5e-690e-487a-8e84-394ffc11f9aa 2024-03-29T06:13:34.447 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:34.444+0000 7f0d1aaf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:34.484 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:34.480+0000 7f0d1aaf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:34.485 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:34.484+0000 7f0d1aaf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:34.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:34.484+0000 7f0d1aaf17c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-29T06:13:34.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:34.484+0000 7f0d1aaf17c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-29T06:13:34.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:34.484+0000 7f0d1aaf17c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-29T06:13:34.636 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:34.632+0000 7f0d1aaf17c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-29T06:13:41.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-29T06:13:41.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:13:41.871 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd7 key to auth repository 2024-03-29T06:13:41.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-29T06:13:41.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:13:42.272 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.7 2024-03-29T06:13:42.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-29T06:13:42.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:13:42.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:13:42.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:13:42.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:13:42.304 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:42.300+0000 7f93b52ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:42.306 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:42.304+0000 7f93b52ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:42.308 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:42.304+0000 7f93b52ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:13:42.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-29T06:13:42.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:13:42.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-29T06:13:42.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:13:42.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:13:42.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:42.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:13:42.578 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:13:42.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:42.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:42.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:42.894 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:42.892+0000 7f93b52ee7c0 -1 Falling back to public interface 2024-03-29T06:13:43.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:43.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:43.884 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:13:43.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:13:43.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:43.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:44.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:45.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:45.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:45.187 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:13:45.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:13:45.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:45.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:45.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:45.752 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:13:45.748+0000 7f93b52ee7c0 -1 osd.7 0 log_to_monitors true 2024-03-29T06:13:46.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:46.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:46.495 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:13:46.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:13:46.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:46.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:46.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:47.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:47.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:47.907 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:13:47.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:13:47.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:47.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:48.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:49.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:49.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:49.303 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:13:49.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:13:49.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:49.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:49.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:50.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:50.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:50.625 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:13:50.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:13:50.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:50.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:51.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:52.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:52.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:52.079 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:13:52.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:13:52.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:52.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:52.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:53.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:53.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:53.420 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:13:53.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:13:53.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:53.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:53.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:54.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:54.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:54.743 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:13:54.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:13:54.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:54.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:55.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:56.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:56.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:56.214 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:13:56.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:13:56.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:56.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:56.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:57.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:57.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:57.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:13:57.667 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:13:57.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:57.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:58.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:13:59.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:13:59.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:13:59.059 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:13:59.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:13:59.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:13:59.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:13:59.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:00.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:00.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:00.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:14:00.442 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:14:00.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:00.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:00.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:01.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:01.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:01.814 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:14:01.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:14:01.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:01.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:02.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:03.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:03.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:03.485 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:14:03.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:14:03.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:03.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:04.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:05.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:05.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:05.163 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:14:05.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:14:05.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:05.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:05.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:06.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:06.911 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:14:06.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:14:06.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:06.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:07.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:08.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:08.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:08.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:14:08.339 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:14:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:09.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:10.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:10.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:10.032 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:14:10.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:14:10.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:10.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:10.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:11.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:11.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:11.434 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:14:11.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:14:11.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:11.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:12.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:13.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:13.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:13.065 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:14:13.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:14:13.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:13.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:13.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:14.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:14.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:14.840 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:14:14.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:14:14.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:14.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:14:15.234 INFO:tasks.workunit.client.0.smithi123.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3538526112,v1:127.0.0.1:6859/3538526112] [v2:127.0.0.1:6860/3538526112,v1:127.0.0.1:6861/3538526112] exists,up efbfff5e-690e-487a-8e84-394ffc11f9aa 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-29T06:14:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:14:15.237 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:14:15.238 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:14:15.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:14:15.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:14:15.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d5e21bb-0422-4128-8cbf-615448408a3a 2024-03-29T06:14:15.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 4d5e21bb-0422-4128-8cbf-615448408a3a' 2024-03-29T06:14:15.241 INFO:tasks.workunit.client.0.smithi123.stdout:add osd8 4d5e21bb-0422-4128-8cbf-615448408a3a 2024-03-29T06:14:15.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:14:15.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3XAZmQPm5DxAA0WlSlqJnwzG/J4/EzSJQPA== 2024-03-29T06:14:15.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3XAZmQPm5DxAA0WlSlqJnwzG/J4/EzSJQPA=="}' 2024-03-29T06:14:15.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d5e21bb-0422-4128-8cbf-615448408a3a -i td/osd-scrub-repair/8/new.json 2024-03-29T06:14:15.870 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:14:15.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-29T06:14:15.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3XAZmQPm5DxAA0WlSlqJnwzG/J4/EzSJQPA== --osd-uuid 4d5e21bb-0422-4128-8cbf-615448408a3a 2024-03-29T06:14:15.934 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:15.931+0000 7fc96fd837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:14:16.022 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:16.019+0000 7fc96fd837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:14:16.023 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:16.019+0000 7fc96fd837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:14:16.024 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:16.019+0000 7fc96fd837c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-29T06:14:16.024 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:16.019+0000 7fc96fd837c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-29T06:14:16.024 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:16.019+0000 7fc96fd837c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-29T06:14:16.108 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:16.103+0000 7fc96fd837c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-29T06:14:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-29T06:14:24.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:14:24.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-29T06:14:24.640 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd8 key to auth repository 2024-03-29T06:14:24.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:14:25.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-29T06:14:25.030 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.8 2024-03-29T06:14:25.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:14:25.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:14:25.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:14:25.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:14:25.061 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:25.055+0000 7ff6524227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:14:25.063 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:25.059+0000 7ff6524227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:14:25.064 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:25.059+0000 7ff6524227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:14:25.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-29T06:14:25.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:14:25.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-29T06:14:25.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:14:25.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:14:25.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:25.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:14:25.331 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:14:25.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:25.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:25.619 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:25.615+0000 7ff6524227c0 -1 Falling back to public interface 2024-03-29T06:14:25.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:26.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:26.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:26.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:14:26.645 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:14:26.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:26.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:26.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:27.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:27.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:27.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:14:27.959 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:14:27.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:27.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:28.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:14:28.643+0000 7ff6524227c0 -1 osd.8 0 log_to_monitors true 2024-03-29T06:14:29.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:29.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:29.290 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:14:29.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:14:29.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:29.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:30.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:31.090 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:14:31.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:14:31.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:31.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:31.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:32.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:32.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:32.514 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:14:32.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:14:32.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:32.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:32.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:33.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:33.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:33.834 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:14:33.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:14:33.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:33.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:34.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:35.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:35.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:35.310 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:14:35.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:14:35.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:35.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:35.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:36.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:14:36.631 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:14:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:36.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:37.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:38.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:38.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:38.130 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:14:38.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:14:38.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:38.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:38.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:39.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:39.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:39.561 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:14:39.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:14:39.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:39.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:39.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:40.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:40.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:40.873 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:14:40.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:14:40.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:40.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:41.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:42.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:42.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:42.298 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:14:42.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:14:42.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:42.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:42.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:43.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:43.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:43.619 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:14:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:14:43.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:43.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:44.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:45.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:45.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:45.023 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:14:45.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:14:45.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:45.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:45.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:46.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:46.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:46.344 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:14:46.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:14:46.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:46.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:46.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:47.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:47.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:47.648 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:14:47.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:14:47.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:47.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:48.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:49.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:49.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:49.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:14:49.067 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:14:49.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:49.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:49.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:50.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:50.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:50.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:14:50.469 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:14:50.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:50.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:51.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:52.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:52.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:52.274 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:14:52.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:14:52.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:52.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:52.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:53.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:53.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:53.801 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:14:53.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:14:53.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:53.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:54.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:55.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:55.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:55.562 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:14:55.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:14:55.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:55.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:56.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:57.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:57.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:57.264 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:14:57.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:14:57.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:57.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:57.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:14:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:14:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:14:58.724 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:14:58.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:14:58.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:14:58.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:14:59.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:00.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:00.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:00.372 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:15:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:15:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:15:00.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:01.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:01.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:01.849 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:15:01.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:15:01.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:01.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:15:02.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:03.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:03.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:03.199 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:15:03.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:15:03.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:03.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:15:03.519 INFO:tasks.workunit.client.0.smithi123.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/467123888,v1:127.0.0.1:6867/467123888] [v2:127.0.0.1:6868/467123888,v1:127.0.0.1:6869/467123888] exists,up 4d5e21bb-0422-4128-8cbf-615448408a3a 2024-03-29T06:15:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:15:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:15:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:15:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:15:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-29T06:15:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:15:03.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:15:03.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:15:03.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:15:03.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:15:03.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:15:03.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:15:03.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:15:03.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:15:03.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:15:03.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:15:03.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 33487da7-fb1a-4701-ab10-066584f8d7a4' 2024-03-29T06:15:03.527 INFO:tasks.workunit.client.0.smithi123.stdout:add osd9 33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:15:03.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:15:03.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnXAZmgvmfIBAAJPK+7wckQdnQdpmi7jiBcw== 2024-03-29T06:15:03.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnXAZmgvmfIBAAJPK+7wckQdnQdpmi7jiBcw=="}' 2024-03-29T06:15:03.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 33487da7-fb1a-4701-ab10-066584f8d7a4 -i td/osd-scrub-repair/9/new.json 2024-03-29T06:15:04.140 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:15:04.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-29T06:15:04.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnXAZmgvmfIBAAJPK+7wckQdnQdpmi7jiBcw== --osd-uuid 33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:15:04.199 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:04.198+0000 7f873389b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:15:04.336 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:04.334+0000 7f873389b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:15:04.338 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:04.338+0000 7f873389b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:15:04.339 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:04.338+0000 7f873389b7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-29T06:15:04.339 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:04.338+0000 7f873389b7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-29T06:15:04.339 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:04.338+0000 7f873389b7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-29T06:15:04.381 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:04.378+0000 7f873389b7c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-29T06:15:12.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-29T06:15:12.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:15:12.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-29T06:15:12.970 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd9 key to auth repository 2024-03-29T06:15:12.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:15:13.351 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.9 2024-03-29T06:15:13.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-29T06:15:13.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:15:13.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:15:13.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:15:13.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:15:13.383 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:13.382+0000 7f10ab2287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:15:13.386 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:13.386+0000 7f10ab2287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:15:13.387 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:13.386+0000 7f10ab2287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:15:13.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-29T06:15:13.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:15:13.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:15:13.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:15:13.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:15:13.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:13.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:15:13.664 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:15:13.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:13.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:13.948 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:13.946+0000 7f10ab2287c0 -1 Falling back to public interface 2024-03-29T06:15:13.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:14.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:14.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:14.972 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:15:14.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:15:14.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:14.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:15.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:16.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:16.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:16.287 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:15:16.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:15:16.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:16.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:16.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:16.836 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:15:16.833+0000 7f10ab2287c0 -1 osd.9 0 log_to_monitors true 2024-03-29T06:15:17.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:17.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:17.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:15:17.608 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:15:17.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:17.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:18.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:19.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:19.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:19.314 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:15:19.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:15:19.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:19.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:19.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:20.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:20.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:20.802 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:15:20.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:15:20.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:20.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:21.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:22.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:22.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:22.258 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:15:22.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:15:22.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:22.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:22.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:23.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:23.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:23.579 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:15:23.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:15:23.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:23.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:24.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:25.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:25.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:25.027 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:15:25.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:15:25.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:25.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:25.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:26.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:26.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:26.348 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:15:26.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:15:26.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:26.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:26.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:27.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:27.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:27.670 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:15:27.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:15:27.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:27.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:28.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:29.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:29.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:29.084 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:15:29.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:15:29.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:29.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:29.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:30.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:30.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:30.497 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:15:30.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:15:30.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:30.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:30.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:31.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:31.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:31.814 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:15:31.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:15:31.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:31.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:32.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:33.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:33.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:33.215 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:15:33.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:15:33.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:33.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:33.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:34.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:34.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:34.538 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:15:34.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:15:34.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:34.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:35.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:36.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:36.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:36.049 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:15:36.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:15:36.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:36.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:36.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:37.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:37.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:37.669 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:15:37.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:15:37.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:37.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:37.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:38.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:38.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:38.989 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:15:38.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:15:38.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:38.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:39.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:40.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:40.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:40.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:15:40.632 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:15:40.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:40.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:41.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:42.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:42.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:42.167 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:15:42.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:15:42.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:42.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:42.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:43.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:43.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:43.897 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:15:43.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:15:43.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:43.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:44.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:45.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:45.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:45.514 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:15:45.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:15:45.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:45.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:46.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:47.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:47.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:47.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:15:47.134 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:15:47.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:47.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:47.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:48.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:48.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:48.856 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:15:48.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:15:48.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:48.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:49.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:50.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:50.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:50.195 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:15:50.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:15:50.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:50.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:50.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:15:51.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:15:51.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:15:51.631 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:15:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:15:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:15:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:15:51.979 INFO:tasks.workunit.client.0.smithi123.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1275345031,v1:127.0.0.1:6875/1275345031] [v2:127.0.0.1:6876/1275345031,v1:127.0.0.1:6877/1275345031] exists,up 33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:15:51.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:15:51.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:15:51.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:15:51.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-29T06:15:51.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:15:52.314 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:15:52.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:15:52.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:15:52.755 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:15:52.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:15:53.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:15:58.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-29T06:15:58.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:15:58.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:15:58.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:15:58.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:15:58.597 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:15:58.597 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:15:58.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:15:58.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:15:58.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:15:58.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:15:58.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:15:58.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:15:58.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:15:58.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:15:58.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:15:59.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:15:59.050 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:15:59.050 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:15:59.050 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:15:59.050 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:15:59.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:15:59.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-03-29T06:15:59.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-03-29T06:15:59.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-03-29T06:15:59.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:15:59.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:15:59.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-29T06:15:59.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-29T06:15:59.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043' 2024-03-29T06:15:59.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:15:59.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:15:59.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-03-29T06:15:59.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-03-29T06:15:59.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508' 2024-03-29T06:15:59.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:15:59.877 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:16:00.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345985 2024-03-29T06:16:00.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345985 2024-03-29T06:16:00.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508 3-85899345985' 2024-03-29T06:16:00.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:00.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:16:00.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182451 2024-03-29T06:16:00.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182451 2024-03-29T06:16:00.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508 3-85899345985 4-107374182451' 2024-03-29T06:16:00.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:00.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:16:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018927 2024-03-29T06:16:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018927 2024-03-29T06:16:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508 3-85899345985 4-107374182451 5-128849018927' 2024-03-29T06:16:00.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:00.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:16:00.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855401 2024-03-29T06:16:00.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855401 2024-03-29T06:16:00.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508 3-85899345985 4-107374182451 5-128849018927 6-150323855401' 2024-03-29T06:16:00.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:00.490 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:16:00.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691867 2024-03-29T06:16:00.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691867 2024-03-29T06:16:00.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508 3-85899345985 4-107374182451 5-128849018927 6-150323855401 7-171798691867' 2024-03-29T06:16:00.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:00.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:16:00.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528339 2024-03-29T06:16:00.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528339 2024-03-29T06:16:00.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508 3-85899345985 4-107374182451 5-128849018927 6-150323855401 7-171798691867 8-193273528339' 2024-03-29T06:16:00.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:00.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:16:00.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364809 2024-03-29T06:16:00.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364809 2024-03-29T06:16:00.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509508 3-85899345985 4-107374182451 5-128849018927 6-150323855401 7-171798691867 8-193273528339 9-214748364809' 2024-03-29T06:16:00.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:00.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-03-29T06:16:00.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:00.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:16:00.905 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:00.905 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-03-29T06:16:00.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-03-29T06:16:00.906 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836569 2024-03-29T06:16:00.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-03-29T06:16:00.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:16:01.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836569 2024-03-29T06:16:01.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:01.195 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-29T06:16:01.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:01.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:16:01.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-29T06:16:01.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:01.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-29T06:16:01.199 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949673043 2024-03-29T06:16:01.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-29T06:16:01.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:16:01.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-03-29T06:16:01.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:01.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-03-29T06:16:01.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:01.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:16:01.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-03-29T06:16:01.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:01.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-03-29T06:16:01.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-03-29T06:16:01.504 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509508 2024-03-29T06:16:01.504 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:16:01.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509508 2024-03-29T06:16:01.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:01.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345985 2024-03-29T06:16:01.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:01.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:16:01.808 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345985 2024-03-29T06:16:01.808 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:01.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345985 2024-03-29T06:16:01.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345985' 2024-03-29T06:16:01.810 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345985 2024-03-29T06:16:01.811 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:16:02.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345985 -lt 85899345985 2024-03-29T06:16:02.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:02.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:02.117 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182451 2024-03-29T06:16:02.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:16:02.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182451 2024-03-29T06:16:02.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:02.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182451 2024-03-29T06:16:02.120 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.4 seq 107374182451 2024-03-29T06:16:02.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-03-29T06:16:02.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:16:02.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182451 -lt 107374182451 2024-03-29T06:16:02.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:02.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018927 2024-03-29T06:16:02.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:02.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:16:02.421 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018927 2024-03-29T06:16:02.421 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:02.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018927 2024-03-29T06:16:02.422 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.5 seq 128849018927 2024-03-29T06:16:02.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018927' 2024-03-29T06:16:02.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:16:02.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018927 -lt 128849018927 2024-03-29T06:16:02.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:02.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:02.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855401 2024-03-29T06:16:02.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:16:02.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855401 2024-03-29T06:16:02.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:02.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855401 2024-03-29T06:16:02.719 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.6 seq 150323855401 2024-03-29T06:16:02.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-03-29T06:16:02.719 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:16:03.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855401 -lt 150323855401 2024-03-29T06:16:03.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:03.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691867 2024-03-29T06:16:03.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:03.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:16:03.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691867 2024-03-29T06:16:03.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:03.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691867 2024-03-29T06:16:03.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691867' 2024-03-29T06:16:03.084 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.7 seq 171798691867 2024-03-29T06:16:03.084 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:16:03.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691867 -lt 171798691867 2024-03-29T06:16:03.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:03.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:03.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528339 2024-03-29T06:16:03.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:16:03.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:03.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528339 2024-03-29T06:16:03.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528339 2024-03-29T06:16:03.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-03-29T06:16:03.421 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.8 seq 193273528339 2024-03-29T06:16:03.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:16:03.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528339 2024-03-29T06:16:03.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:03.708 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364809 2024-03-29T06:16:03.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:03.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:16:03.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364809 2024-03-29T06:16:03.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:03.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364809 2024-03-29T06:16:03.712 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.9 seq 214748364809 2024-03-29T06:16:03.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364809' 2024-03-29T06:16:03.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:16:04.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364810 -lt 214748364809 2024-03-29T06:16:04.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:16:04.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:04.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:04.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:16:04.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:16:04.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:16:04.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:16:04.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:16:04.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:16:04.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:16:04.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:16:04.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:16:04.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:16:04.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:04.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T06:16:05.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-29T06:16:05.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T06:16:05.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T06:16:06.679 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T06:16:06.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:16:07.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-29T06:16:07.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T06:16:07.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:16:07.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:16:07.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:16:07.708 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:16:07.708 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:16:07.709 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:16:07.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:16:07.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:16:07.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:16:07.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:16:07.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:16:07.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:16:07.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:16:07.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:16:07.841 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:16:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:16:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:16:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:16:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:16:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:16:08.339 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:16:08.340 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:16:08.340 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:16:08.340 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:16:08.340 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:16:08.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:16:08.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:08.340 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:16:08.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-03-29T06:16:08.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-03-29T06:16:08.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-03-29T06:16:08.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:08.543 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:16:08.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-03-29T06:16:08.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-03-29T06:16:08.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046' 2024-03-29T06:16:08.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:08.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:16:08.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-03-29T06:16:08.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-03-29T06:16:08.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510' 2024-03-29T06:16:08.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:08.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:16:09.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345988 2024-03-29T06:16:09.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345988 2024-03-29T06:16:09.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510 3-85899345988' 2024-03-29T06:16:09.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:09.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:16:09.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182454 2024-03-29T06:16:09.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182454 2024-03-29T06:16:09.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510 3-85899345988 4-107374182454' 2024-03-29T06:16:09.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:09.230 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:16:09.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018930 2024-03-29T06:16:09.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018930 2024-03-29T06:16:09.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510 3-85899345988 4-107374182454 5-128849018930' 2024-03-29T06:16:09.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:09.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:16:09.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855404 2024-03-29T06:16:09.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855404 2024-03-29T06:16:09.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510 3-85899345988 4-107374182454 5-128849018930 6-150323855404' 2024-03-29T06:16:09.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:09.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:16:09.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691870 2024-03-29T06:16:09.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691870 2024-03-29T06:16:09.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510 3-85899345988 4-107374182454 5-128849018930 6-150323855404 7-171798691870' 2024-03-29T06:16:09.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:09.784 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:16:09.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528342 2024-03-29T06:16:09.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528342 2024-03-29T06:16:09.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510 3-85899345988 4-107374182454 5-128849018930 6-150323855404 7-171798691870 8-193273528342' 2024-03-29T06:16:09.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:09.935 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:16:10.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-03-29T06:16:10.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-03-29T06:16:10.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673046 2-64424509510 3-85899345988 4-107374182454 5-128849018930 6-150323855404 7-171798691870 8-193273528342 9-214748364812' 2024-03-29T06:16:10.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:10.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-03-29T06:16:10.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:16:10.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-03-29T06:16:10.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:10.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-03-29T06:16:10.077 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836572 2024-03-29T06:16:10.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-03-29T06:16:10.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:16:10.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-03-29T06:16:10.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:10.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-03-29T06:16:10.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:10.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:16:10.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-03-29T06:16:10.378 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:10.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-03-29T06:16:10.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-03-29T06:16:10.380 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949673046 2024-03-29T06:16:10.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:16:10.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-03-29T06:16:10.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:10.677 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-03-29T06:16:10.677 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:10.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:16:10.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:10.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-03-29T06:16:10.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-03-29T06:16:10.681 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509510 2024-03-29T06:16:10.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-03-29T06:16:10.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:16:11.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509510 2024-03-29T06:16:11.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:11.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345988 2024-03-29T06:16:11.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:11.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:16:11.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345988 2024-03-29T06:16:11.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:11.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345988 2024-03-29T06:16:11.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345988' 2024-03-29T06:16:11.050 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 85899345988 2024-03-29T06:16:11.050 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:16:11.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345988 -lt 85899345988 2024-03-29T06:16:11.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:11.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182454 2024-03-29T06:16:11.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:11.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:16:11.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182454 2024-03-29T06:16:11.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:11.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182454 2024-03-29T06:16:11.435 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.4 seq 107374182454 2024-03-29T06:16:11.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-03-29T06:16:11.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:16:11.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182454 -lt 107374182454 2024-03-29T06:16:11.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:11.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018930 2024-03-29T06:16:11.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:11.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:16:11.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018930 2024-03-29T06:16:11.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:11.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018930 2024-03-29T06:16:11.753 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.5 seq 128849018930 2024-03-29T06:16:11.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018930' 2024-03-29T06:16:11.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:16:12.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018930 -lt 128849018930 2024-03-29T06:16:12.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:12.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855404 2024-03-29T06:16:12.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:12.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:16:12.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855404 2024-03-29T06:16:12.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855404 2024-03-29T06:16:12.071 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.6 seq 150323855404 2024-03-29T06:16:12.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855404' 2024-03-29T06:16:12.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:16:12.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855404 -lt 150323855404 2024-03-29T06:16:12.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:12.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691870 2024-03-29T06:16:12.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:12.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:16:12.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691870 2024-03-29T06:16:12.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:12.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691870 2024-03-29T06:16:12.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-03-29T06:16:12.433 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.7 seq 171798691870 2024-03-29T06:16:12.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:16:12.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691870 -lt 171798691870 2024-03-29T06:16:12.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:12.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528342 2024-03-29T06:16:12.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:12.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:16:12.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:12.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528342 2024-03-29T06:16:12.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528342 2024-03-29T06:16:12.730 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.8 seq 193273528342 2024-03-29T06:16:12.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528342' 2024-03-29T06:16:12.730 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:16:13.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528342 -lt 193273528342 2024-03-29T06:16:13.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:13.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-03-29T06:16:13.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:13.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:16:13.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-03-29T06:16:13.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:13.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-03-29T06:16:13.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-03-29T06:16:13.073 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.9 seq 214748364812 2024-03-29T06:16:13.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:16:13.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364812 -lt 214748364812 2024-03-29T06:16:13.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:16:13.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:13.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:13.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:16:13.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:16:13.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:16:13.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:16:13.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:16:13.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:16:13.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:16:13.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:16:14.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:16:14.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:16:14.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:14.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:14.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:16:14.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:16:14.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:16:14.872 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:16:14.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:16:15.374 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:16:15.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:16:15.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:16:15.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T06:16:16.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-29T06:16:16.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T06:16:16.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T06:16:16.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:16:16.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:16:16.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-29T06:16:16.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-29T06:16:16.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T06:16:16.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-29T06:16:16.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-29T06:16:16.578 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:16:16.578 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:16:16.892 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-29T06:16:16.892 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:16:16.893 INFO:tasks.workunit.client.0.smithi123.stderr:9 2024-03-29T06:16:16.893 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:16:16.893 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:16:16.893 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:16:16.893 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:16:16.893 INFO:tasks.workunit.client.0.smithi123.stderr:7' 2024-03-29T06:16:16.893 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:16:16.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:16:16.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:16:16.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:16:16.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:16:18.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:16:18.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:16:18.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:16:18.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:16:18.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:16:18.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:16:18.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:16:18.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:16:20.653 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:16:21.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:16:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:16:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:16:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:16:21.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:16:21.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:16:21.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:16:21.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:16:21.245 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:21.240+0000 7f3f6c3617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:21.247 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:21.244+0000 7f3f6c3617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:21.249 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:21.248+0000 7f3f6c3617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:21.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:16:21.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:16:21.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:16:21.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:16:21.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:16:21.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:21.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:16:21.530 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:16:21.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:21.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:16:21.795 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:21.792+0000 7f3f6c3617c0 -1 Falling back to public interface 2024-03-29T06:16:21.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:22.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:22.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:22.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:16:22.845 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:16:22.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:22.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:16:23.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:24.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:24.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:24.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:16:24.150 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:16:24.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:24.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:16:24.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:24.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:24.908+0000 7f3f6c3617c0 -1 osd.3 66 log_to_monitors true 2024-03-29T06:16:25.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:25.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:25.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:16:25.456 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:16:25.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:25.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:16:26.013 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 70 up_thru 62 down_at 67 last_clean_interval [20,66) [v2:127.0.0.1:6826/1303768558,v1:127.0.0.1:6827/1303768558] [v2:127.0.0.1:6828/1303768558,v1:127.0.0.1:6829/1303768558] exists,up 75727be0-e71a-4f24-99d4-cdaee9b3727e 2024-03-29T06:16:26.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:16:26.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:16:26.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:16:26.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:16:26.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:16:26.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:16:26.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:16:26.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:16:26.014 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:16:26.014 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:16:26.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:16:26.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:16:26.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:16:26.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:16:26.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:16:26.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:16:26.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:16:26.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:16:26.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:16:26.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:26.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:16:26.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-03-29T06:16:26.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-03-29T06:16:26.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-03-29T06:16:26.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:26.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:16:27.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-03-29T06:16:27.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-03-29T06:16:27.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050' 2024-03-29T06:16:27.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:27.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:16:27.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-03-29T06:16:27.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-03-29T06:16:27.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515' 2024-03-29T06:16:27.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:27.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:16:27.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710722 2024-03-29T06:16:27.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710722 2024-03-29T06:16:27.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515 3-300647710722' 2024-03-29T06:16:27.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:27.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:16:27.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182458 2024-03-29T06:16:27.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182458 2024-03-29T06:16:27.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515 3-300647710722 4-107374182458' 2024-03-29T06:16:27.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:27.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:16:27.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018934 2024-03-29T06:16:27.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018934 2024-03-29T06:16:27.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515 3-300647710722 4-107374182458 5-128849018934' 2024-03-29T06:16:27.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:27.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:16:28.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855408 2024-03-29T06:16:28.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855408 2024-03-29T06:16:28.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515 3-300647710722 4-107374182458 5-128849018934 6-150323855408' 2024-03-29T06:16:28.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:28.131 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:16:28.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691874 2024-03-29T06:16:28.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691874 2024-03-29T06:16:28.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515 3-300647710722 4-107374182458 5-128849018934 6-150323855408 7-171798691874' 2024-03-29T06:16:28.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:28.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:16:28.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528347 2024-03-29T06:16:28.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528347 2024-03-29T06:16:28.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515 3-300647710722 4-107374182458 5-128849018934 6-150323855408 7-171798691874 8-193273528347' 2024-03-29T06:16:28.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:28.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:16:28.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364817 2024-03-29T06:16:28.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364817 2024-03-29T06:16:28.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509515 3-300647710722 4-107374182458 5-128849018934 6-150323855408 7-171798691874 8-193273528347 9-214748364817' 2024-03-29T06:16:28.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:28.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-03-29T06:16:28.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:28.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:16:28.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-03-29T06:16:28.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:28.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-03-29T06:16:28.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-03-29T06:16:28.594 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836577 2024-03-29T06:16:28.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:16:28.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836577 2024-03-29T06:16:28.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:28.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-03-29T06:16:28.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:28.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:16:28.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-03-29T06:16:28.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-03-29T06:16:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-03-29T06:16:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673050 2024-03-29T06:16:28.909 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:16:29.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-03-29T06:16:29.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:29.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-03-29T06:16:29.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:29.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:16:29.213 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-03-29T06:16:29.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:29.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-03-29T06:16:29.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-03-29T06:16:29.215 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509515 2024-03-29T06:16:29.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:16:29.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509515 2024-03-29T06:16:29.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:29.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:29.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710722 2024-03-29T06:16:29.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:16:29.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710722 2024-03-29T06:16:29.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:29.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710722 2024-03-29T06:16:29.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710722' 2024-03-29T06:16:29.546 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 300647710722 2024-03-29T06:16:29.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:16:29.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710722 -lt 300647710722 2024-03-29T06:16:29.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:29.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182458 2024-03-29T06:16:29.843 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:29.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:16:29.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182458 2024-03-29T06:16:29.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:29.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182458 2024-03-29T06:16:29.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-03-29T06:16:29.847 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 107374182458 2024-03-29T06:16:29.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:16:30.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182459 -lt 107374182458 2024-03-29T06:16:30.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:30.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018934 2024-03-29T06:16:30.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:30.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:16:30.205 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018934 2024-03-29T06:16:30.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:30.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018934 2024-03-29T06:16:30.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018934' 2024-03-29T06:16:30.207 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 128849018934 2024-03-29T06:16:30.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:16:30.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018935 -lt 128849018934 2024-03-29T06:16:30.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:30.518 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855408 2024-03-29T06:16:30.518 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:30.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:16:30.520 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855408 2024-03-29T06:16:30.520 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:30.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855408 2024-03-29T06:16:30.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-03-29T06:16:30.521 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 150323855408 2024-03-29T06:16:30.522 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:16:30.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855409 -lt 150323855408 2024-03-29T06:16:30.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:30.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691874 2024-03-29T06:16:30.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:30.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:16:30.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691874 2024-03-29T06:16:30.833 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:30.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691874 2024-03-29T06:16:30.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-03-29T06:16:30.835 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 171798691874 2024-03-29T06:16:30.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:16:31.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691874 2024-03-29T06:16:31.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:31.136 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528347 2024-03-29T06:16:31.136 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:31.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:16:31.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528347 2024-03-29T06:16:31.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:31.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528347 2024-03-29T06:16:31.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2024-03-29T06:16:31.140 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 193273528347 2024-03-29T06:16:31.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:16:31.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528347 -lt 193273528347 2024-03-29T06:16:31.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:31.507 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364817 2024-03-29T06:16:31.507 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:31.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:16:31.510 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364817 2024-03-29T06:16:31.510 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:31.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364817 2024-03-29T06:16:31.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-03-29T06:16:31.512 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 214748364817 2024-03-29T06:16:31.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:16:31.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364817 -lt 214748364817 2024-03-29T06:16:31.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:16:31.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:31.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:32.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:16:32.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:16:32.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:16:32.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:16:32.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:16:32.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:16:32.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:16:32.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:16:32.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:16:32.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:16:32.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:32.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:32.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:16:32.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:16:32.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:16:32.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:16:32.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:16:32.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:16:32.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:16:32.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:16:33.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:16:33.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:16:33.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:16:33.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:16:33.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:16:33.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:16:33.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:16:33.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:16:33.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:16:06.386511+0000 2024-03-29T06:16:33.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:16:33.863 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:16:06.386511+0000 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:16:06.386511+0000 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:16:33.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:16:33.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:16:33.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:16:33.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:16:34.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:16:06.386511+0000 '>' 2024-03-29T06:16:06.386511+0000 2024-03-29T06:16:34.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:16:35.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:16:35.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:16:35.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:16:35.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:16:35.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:16:35.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:16:35.413 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:16:35.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:16:33.976621+0000 '>' 2024-03-29T06:16:06.386511+0000 2024-03-29T06:16:35.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:16:35.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:16:35.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:16:35.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:16:35.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:16:36.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:16:36.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:16:36.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:16:36.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:16:36.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:16:36.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:16:36.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:16:36.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-29T06:16:38.924 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:16:38.924 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:16:38.924 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:16:39.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:16:39.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:16:39.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:16:39.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:16:39.216 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:16:39.217 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:16:39.217 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:16:39.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:16:39.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:16:39.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:16:39.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:16:39.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:16:39.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.9 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:16:39.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:16:39.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:16:39.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:16:39.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:16:39.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:16:39.251 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:39.247+0000 7f4249dea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:39.254 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:39.251+0000 7f4249dea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:39.255 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:39.251+0000 7f4249dea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:39.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:39.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:16:39.792 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:39.787+0000 7f4249dea7c0 -1 Falling back to public interface 2024-03-29T06:16:39.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:40.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:40.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:40.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:16:40.922 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:16:40.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:40.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:16:41.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:42.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:42.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:42.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:16:42.243 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:16:42.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:42.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:16:42.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:42.868 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:42.863+0000 7f4249dea7c0 -1 osd.9 71 log_to_monitors true 2024-03-29T06:16:43.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:43.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:43.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:16:43.567 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:16:43.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:43.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:16:44.143 INFO:tasks.workunit.client.0.smithi123.stderr:osd.9 up in weight 1 up_from 75 up_thru 0 down_at 72 last_clean_interval [50,71) [v2:127.0.0.1:6874/3606071363,v1:127.0.0.1:6875/3606071363] [v2:127.0.0.1:6876/3606071363,v1:127.0.0.1:6877/3606071363] exists,up 33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:16:44.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:16:44.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:16:44.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:16:44.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:16:44.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:16:44.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:16:44.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:16:44.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:16:44.144 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:16:44.144 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:16:44.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:16:44.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:16:44.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:16:44.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:16:44.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:16:44.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:16:44.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:16:44.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:16:44.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:16:44.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:16:44.766 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:16:44.766 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:44.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:16:44.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-03-29T06:16:44.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-03-29T06:16:44.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-03-29T06:16:44.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:44.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:16:45.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-03-29T06:16:45.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-03-29T06:16:45.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055' 2024-03-29T06:16:45.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:45.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:16:45.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-03-29T06:16:45.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-03-29T06:16:45.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520' 2024-03-29T06:16:45.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:45.538 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:16:46.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710727 2024-03-29T06:16:46.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710727 2024-03-29T06:16:46.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520 3-300647710727' 2024-03-29T06:16:46.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:46.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:16:46.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-03-29T06:16:46.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-03-29T06:16:46.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520 3-300647710727 4-107374182463' 2024-03-29T06:16:46.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:46.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:16:46.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018939 2024-03-29T06:16:46.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018939 2024-03-29T06:16:46.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520 3-300647710727 4-107374182463 5-128849018939' 2024-03-29T06:16:46.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:46.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:16:46.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-03-29T06:16:46.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-03-29T06:16:46.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520 3-300647710727 4-107374182463 5-128849018939 6-150323855413' 2024-03-29T06:16:46.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:46.694 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:16:46.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-03-29T06:16:46.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-03-29T06:16:46.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520 3-300647710727 4-107374182463 5-128849018939 6-150323855413 7-171798691879' 2024-03-29T06:16:46.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:46.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:16:46.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528351 2024-03-29T06:16:46.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528351 2024-03-29T06:16:46.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520 3-300647710727 4-107374182463 5-128849018939 6-150323855413 7-171798691879 8-193273528351' 2024-03-29T06:16:46.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:16:46.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:16:47.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-03-29T06:16:47.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-03-29T06:16:47.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509520 3-300647710727 4-107374182463 5-128849018939 6-150323855413 7-171798691879 8-193273528351 9-322122547202' 2024-03-29T06:16:47.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:47.120 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-03-29T06:16:47.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:47.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:16:47.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-03-29T06:16:47.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:47.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-03-29T06:16:47.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-03-29T06:16:47.124 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836581 2024-03-29T06:16:47.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:16:47.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836581 2024-03-29T06:16:47.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:47.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-03-29T06:16:47.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:47.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:16:47.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-03-29T06:16:47.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:47.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-03-29T06:16:47.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-03-29T06:16:47.427 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673055 2024-03-29T06:16:47.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:16:47.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673055 2024-03-29T06:16:47.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:47.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-03-29T06:16:47.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:47.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:16:47.731 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-03-29T06:16:47.731 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:47.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-03-29T06:16:47.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-03-29T06:16:47.733 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509520 2024-03-29T06:16:47.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:16:48.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509520 2024-03-29T06:16:48.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:48.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710727 2024-03-29T06:16:48.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:48.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:16:48.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710727 2024-03-29T06:16:48.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:48.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710727 2024-03-29T06:16:48.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710727' 2024-03-29T06:16:48.030 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 300647710727 2024-03-29T06:16:48.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:16:48.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710727 -lt 300647710727 2024-03-29T06:16:48.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:48.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-03-29T06:16:48.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:48.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:16:48.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-03-29T06:16:48.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:48.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-03-29T06:16:48.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-03-29T06:16:48.367 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 107374182463 2024-03-29T06:16:48.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:16:48.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182463 2024-03-29T06:16:48.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:48.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018939 2024-03-29T06:16:48.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:48.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:16:48.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018939 2024-03-29T06:16:48.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:48.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018939 2024-03-29T06:16:48.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018939' 2024-03-29T06:16:48.662 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 128849018939 2024-03-29T06:16:48.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:16:48.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018939 -lt 128849018939 2024-03-29T06:16:48.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:48.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-03-29T06:16:48.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:48.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:16:48.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-03-29T06:16:48.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:48.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-03-29T06:16:48.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-03-29T06:16:48.980 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 150323855413 2024-03-29T06:16:48.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:16:49.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855413 2024-03-29T06:16:49.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:49.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-03-29T06:16:49.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:49.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:16:49.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-03-29T06:16:49.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:49.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-03-29T06:16:49.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-03-29T06:16:49.274 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 171798691879 2024-03-29T06:16:49.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:16:49.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691879 2024-03-29T06:16:49.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:49.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528351 2024-03-29T06:16:49.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:49.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:16:49.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:49.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528351 2024-03-29T06:16:49.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528351 2024-03-29T06:16:49.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-03-29T06:16:49.566 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 193273528351 2024-03-29T06:16:49.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:16:49.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528352 -lt 193273528351 2024-03-29T06:16:49.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:16:49.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-322122547202 2024-03-29T06:16:49.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:16:49.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:16:49.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:16:49.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-322122547202 2024-03-29T06:16:49.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-03-29T06:16:49.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 322122547202' 2024-03-29T06:16:49.920 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 322122547202 2024-03-29T06:16:49.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:16:50.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-03-29T06:16:50.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:16:50.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:50.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:50.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:16:50.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:16:50.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:16:50.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:16:50.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:16:50.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:16:50.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:16:50.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:16:50.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:16:50.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:16:50.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:16:50.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:16:51.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:16:51.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:16:51.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:16:51.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:16:51.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:16:51.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-29T06:16:51.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:16:51.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:16:51.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-29T06:16:51.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:16:51.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:16:51.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:16:51.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:16:51.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:16:51.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:16:51.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:16:51.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:16:51.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:16:51.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:16:51.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:16:51.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:16:51.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:16:51.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:16:51.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:16:51.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:16:52.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-29T06:16:54.737 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:16:55.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:16:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:16:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:16:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:16:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:16:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:16:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:16:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:16:55.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:16:55.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:16:55.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:16:55.282 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:16:55.282 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:16:55.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:16:55.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:16:55.284 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.5 2024-03-29T06:16:55.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:16:55.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:16:55.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:16:55.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:16:55.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:16:55.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:16:55.319 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:55.315+0000 7f520ab0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:55.321 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:55.319+0000 7f520ab0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:55.323 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:55.319+0000 7f520ab0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:55.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:16:55.860 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:55.855+0000 7f520ab0a7c0 -1 Falling back to public interface 2024-03-29T06:16:55.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:56.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:56.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:56.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:16:56.931 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:16:56.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:56.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:16:57.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:58.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:58.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:58.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:16:58.241 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:16:58.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:58.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:16:58.536 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:16:58.531+0000 7f520ab0a7c0 -1 osd.5 77 log_to_monitors true 2024-03-29T06:16:58.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:16:59.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:16:59.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:16:59.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:16:59.558 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:16:59.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:16:59.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:17:00.026 INFO:tasks.workunit.client.0.smithi123.stderr:osd.5 up in weight 1 up_from 81 up_thru 81 down_at 78 last_clean_interval [30,77) [v2:127.0.0.1:6842/3286980358,v1:127.0.0.1:6843/3286980358] [v2:127.0.0.1:6844/3286980358,v1:127.0.0.1:6845/3286980358] exists,up 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:17:00.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:17:00.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:17:00.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:17:00.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:17:00.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:17:00.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:17:00.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:17:00.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:17:00.029 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:17:00.029 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:17:00.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:17:00.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:17:00.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:17:00.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:17:00.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:17:00.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:17:00.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:17:00.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:17:00.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:17:00.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:17:00.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:00.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:17:00.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836585 2024-03-29T06:17:00.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836585 2024-03-29T06:17:00.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585' 2024-03-29T06:17:00.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:00.932 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:17:01.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-03-29T06:17:01.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-03-29T06:17:01.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059' 2024-03-29T06:17:01.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:01.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:17:01.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-03-29T06:17:01.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-03-29T06:17:01.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524' 2024-03-29T06:17:01.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:01.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:17:01.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710731 2024-03-29T06:17:01.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710731 2024-03-29T06:17:01.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524 3-300647710731' 2024-03-29T06:17:01.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:01.535 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:17:01.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-03-29T06:17:01.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-03-29T06:17:01.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524 3-300647710731 4-107374182467' 2024-03-29T06:17:01.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:01.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:17:02.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-03-29T06:17:02.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-03-29T06:17:02.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524 3-300647710731 4-107374182467 5-347892350978' 2024-03-29T06:17:02.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:02.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:17:02.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855417 2024-03-29T06:17:02.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855417 2024-03-29T06:17:02.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524 3-300647710731 4-107374182467 5-347892350978 6-150323855417' 2024-03-29T06:17:02.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:02.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:17:02.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-03-29T06:17:02.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-03-29T06:17:02.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524 3-300647710731 4-107374182467 5-347892350978 6-150323855417 7-171798691883' 2024-03-29T06:17:02.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:02.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:17:02.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528355 2024-03-29T06:17:02.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528355 2024-03-29T06:17:02.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524 3-300647710731 4-107374182467 5-347892350978 6-150323855417 7-171798691883 8-193273528355' 2024-03-29T06:17:02.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:02.490 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:17:02.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547206 2024-03-29T06:17:02.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547206 2024-03-29T06:17:02.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509524 3-300647710731 4-107374182467 5-347892350978 6-150323855417 7-171798691883 8-193273528355 9-322122547206' 2024-03-29T06:17:02.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:02.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836585 2024-03-29T06:17:02.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:02.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:17:02.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836585 2024-03-29T06:17:02.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:02.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836585 2024-03-29T06:17:02.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-03-29T06:17:02.635 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836585 2024-03-29T06:17:02.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:17:02.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836585 2024-03-29T06:17:02.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:02.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:02.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-03-29T06:17:02.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:17:02.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-03-29T06:17:02.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:02.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-03-29T06:17:02.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-03-29T06:17:02.948 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673059 2024-03-29T06:17:02.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:17:03.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-03-29T06:17:03.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:03.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-03-29T06:17:03.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:03.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:17:03.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-03-29T06:17:03.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:03.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-03-29T06:17:03.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-03-29T06:17:03.264 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509524 2024-03-29T06:17:03.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:17:03.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509524 2024-03-29T06:17:03.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:03.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710731 2024-03-29T06:17:03.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:03.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:17:03.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710731 2024-03-29T06:17:03.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:03.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710731 2024-03-29T06:17:03.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710731' 2024-03-29T06:17:03.642 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 300647710731 2024-03-29T06:17:03.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:17:03.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710731 -lt 300647710731 2024-03-29T06:17:03.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:03.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-03-29T06:17:03.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:03.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:17:03.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-03-29T06:17:03.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:03.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-03-29T06:17:03.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-03-29T06:17:03.972 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 107374182467 2024-03-29T06:17:03.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:17:04.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182467 -lt 107374182467 2024-03-29T06:17:04.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:04.318 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-347892350978 2024-03-29T06:17:04.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:04.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:17:04.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-347892350978 2024-03-29T06:17:04.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:04.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-03-29T06:17:04.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 347892350978' 2024-03-29T06:17:04.322 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 347892350978 2024-03-29T06:17:04.323 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:17:04.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-03-29T06:17:04.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:04.694 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855417 2024-03-29T06:17:04.694 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:04.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:17:04.696 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855417 2024-03-29T06:17:04.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:04.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855417 2024-03-29T06:17:04.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-03-29T06:17:04.698 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 150323855417 2024-03-29T06:17:04.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:17:05.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855417 2024-03-29T06:17:05.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:05.037 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-03-29T06:17:05.037 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:05.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:17:05.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-03-29T06:17:05.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:05.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-03-29T06:17:05.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-03-29T06:17:05.040 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 171798691883 2024-03-29T06:17:05.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:17:05.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691883 -lt 171798691883 2024-03-29T06:17:05.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:05.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528355 2024-03-29T06:17:05.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:05.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:17:05.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528355 2024-03-29T06:17:05.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:05.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528355 2024-03-29T06:17:05.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-03-29T06:17:05.438 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 193273528355 2024-03-29T06:17:05.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:17:05.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528356 -lt 193273528355 2024-03-29T06:17:05.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:05.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-322122547206 2024-03-29T06:17:05.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:05.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:17:05.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-322122547206 2024-03-29T06:17:05.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:05.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547206 2024-03-29T06:17:05.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 322122547206' 2024-03-29T06:17:05.767 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 322122547206 2024-03-29T06:17:05.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:17:06.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547207 -lt 322122547206 2024-03-29T06:17:06.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:17:06.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:06.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:06.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:17:06.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:17:06.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:17:06.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:17:06.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:17:06.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:17:06.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:17:06.466 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:17:06.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:17:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:17:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:07.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:17:07.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:17:07.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:17:07.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:17:07.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:17:07.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:17:07.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:17:07.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:17:07.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:17:07.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:17:07.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:17:07.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:17:07.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:07.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:07.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:07.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:07.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:16:33.976621+0000 2024-03-29T06:17:07.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:17:08.064 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:17:08.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:16:33.976621+0000 2024-03-29T06:17:08.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:17:08.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:16:33.976621+0000 2024-03-29T06:17:08.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:17:08.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:17:08.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:17:08.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:17:08.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:08.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:08.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:08.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:08.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:16:33.976621+0000 '>' 2024-03-29T06:16:33.976621+0000 2024-03-29T06:17:08.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:17:09.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:17:09.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:17:09.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:17:09.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:09.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:09.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:09.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:09.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:16:33.976621+0000 '>' 2024-03-29T06:16:33.976621+0000 2024-03-29T06:17:09.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:17:10.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:17:10.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:17:10.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:17:10.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:10.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:10.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:10.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:11.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:16:33.976621+0000 '>' 2024-03-29T06:16:33.976621+0000 2024-03-29T06:17:11.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:17:12.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:17:12.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:17:12.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:17:12.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:12.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:12.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:12.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:08.920172+0000 '>' 2024-03-29T06:16:33.976621+0000 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:17:12.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:17:12.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:17:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:17:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:17:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:17:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:17:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:17:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:17:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:17:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-29T06:17:15.413 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:17:15.413 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:17:15.413 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:17:15.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:17:15.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:17:15.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:17:15.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:17:15.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:17:15.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:17:15.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:17:15.737 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.9 2024-03-29T06:17:15.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:17:15.737 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:17:15.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:17:15.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:17:15.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:17:15.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:17:15.770 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:17:15.766+0000 7f14e13827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:15.773 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:17:15.770+0000 7f14e13827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:15.774 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:17:15.770+0000 7f14e13827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:16.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:17:16.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:17:16.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:17:16.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:17:16.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:17:16.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:16.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:17:16.082 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:17:16.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:16.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:16.311 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:17:16.306+0000 7f14e13827c0 -1 Falling back to public interface 2024-03-29T06:17:16.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:17.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:17.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:17.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:17:17.465 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:17:17.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:17.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:17.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:18.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:17:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:17:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:19.019 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:17:19.014+0000 7f14e13827c0 -1 osd.9 83 log_to_monitors true 2024-03-29T06:17:19.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:20.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:20.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:20.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:17:20.160 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:17:20.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:20.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:20.925 INFO:tasks.workunit.client.0.smithi123.stderr:osd.9 up in weight 1 up_from 87 up_thru 0 down_at 84 last_clean_interval [75,83) [v2:127.0.0.1:6874/1557156583,v1:127.0.0.1:6875/1557156583] [v2:127.0.0.1:6876/1557156583,v1:127.0.0.1:6877/1557156583] exists,up 33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:17:20.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:17:20.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:17:20.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:17:20.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:17:20.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:17:20.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:17:20.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:17:20.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:17:20.926 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:17:20.926 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:17:20.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:17:20.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:17:20.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:17:21.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:17:21.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:17:21.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:17:21.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:17:21.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:17:21.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:17:21.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:17:21.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:21.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:17:22.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836591 2024-03-29T06:17:22.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836591 2024-03-29T06:17:22.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591' 2024-03-29T06:17:22.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:22.273 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:17:22.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673064 2024-03-29T06:17:22.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673064 2024-03-29T06:17:22.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064' 2024-03-29T06:17:22.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:22.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:17:23.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-03-29T06:17:23.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-03-29T06:17:23.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529' 2024-03-29T06:17:23.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:23.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:17:23.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710736 2024-03-29T06:17:23.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710736 2024-03-29T06:17:23.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529 3-300647710736' 2024-03-29T06:17:23.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:23.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:17:23.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-03-29T06:17:23.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-03-29T06:17:23.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529 3-300647710736 4-107374182472' 2024-03-29T06:17:23.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:23.295 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:17:23.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350983 2024-03-29T06:17:23.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350983 2024-03-29T06:17:23.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529 3-300647710736 4-107374182472 5-347892350983' 2024-03-29T06:17:23.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:23.487 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:17:23.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855422 2024-03-29T06:17:23.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855422 2024-03-29T06:17:23.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529 3-300647710736 4-107374182472 5-347892350983 6-150323855422' 2024-03-29T06:17:23.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:23.629 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:17:23.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691888 2024-03-29T06:17:23.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691888 2024-03-29T06:17:23.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529 3-300647710736 4-107374182472 5-347892350983 6-150323855422 7-171798691888' 2024-03-29T06:17:23.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:23.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:17:23.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528361 2024-03-29T06:17:23.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528361 2024-03-29T06:17:23.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529 3-300647710736 4-107374182472 5-347892350983 6-150323855422 7-171798691888 8-193273528361' 2024-03-29T06:17:23.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:23.942 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:17:24.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154754 2024-03-29T06:17:24.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154754 2024-03-29T06:17:24.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509529 3-300647710736 4-107374182472 5-347892350983 6-150323855422 7-171798691888 8-193273528361 9-373662154754' 2024-03-29T06:17:24.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:24.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836591 2024-03-29T06:17:24.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:24.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:17:24.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836591 2024-03-29T06:17:24.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:24.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836591 2024-03-29T06:17:24.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-03-29T06:17:24.088 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836591 2024-03-29T06:17:24.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:17:24.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836591 -lt 21474836591 2024-03-29T06:17:24.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:24.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673064 2024-03-29T06:17:24.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:24.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:17:24.390 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673064 2024-03-29T06:17:24.390 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:24.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673064 2024-03-29T06:17:24.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-03-29T06:17:24.392 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673064 2024-03-29T06:17:24.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:17:24.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673065 -lt 42949673064 2024-03-29T06:17:24.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:24.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-03-29T06:17:24.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:24.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:17:24.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-03-29T06:17:24.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:24.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-03-29T06:17:24.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-03-29T06:17:24.709 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509529 2024-03-29T06:17:24.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:17:25.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509529 -lt 64424509529 2024-03-29T06:17:25.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:25.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:25.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710736 2024-03-29T06:17:25.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:17:25.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710736 2024-03-29T06:17:25.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:25.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710736 2024-03-29T06:17:25.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710736' 2024-03-29T06:17:25.006 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 300647710736 2024-03-29T06:17:25.006 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:17:25.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710736 -lt 300647710736 2024-03-29T06:17:25.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:25.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:25.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-03-29T06:17:25.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:17:25.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-03-29T06:17:25.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:25.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-03-29T06:17:25.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-03-29T06:17:25.309 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 107374182472 2024-03-29T06:17:25.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:17:25.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182472 2024-03-29T06:17:25.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:25.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-347892350983 2024-03-29T06:17:25.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:25.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:17:25.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-347892350983 2024-03-29T06:17:25.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:25.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350983 2024-03-29T06:17:25.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 347892350983' 2024-03-29T06:17:25.628 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 347892350983 2024-03-29T06:17:25.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:17:25.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350984 -lt 347892350983 2024-03-29T06:17:25.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:25.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855422 2024-03-29T06:17:25.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:25.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:17:25.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:25.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855422 2024-03-29T06:17:25.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855422 2024-03-29T06:17:25.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-03-29T06:17:25.928 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 150323855422 2024-03-29T06:17:25.928 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:17:26.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855423 -lt 150323855422 2024-03-29T06:17:26.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:26.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691888 2024-03-29T06:17:26.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:26.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:17:26.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691888 2024-03-29T06:17:26.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:26.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691888 2024-03-29T06:17:26.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-03-29T06:17:26.281 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 171798691888 2024-03-29T06:17:26.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:17:26.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691889 -lt 171798691888 2024-03-29T06:17:26.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:26.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528361 2024-03-29T06:17:26.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:26.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:17:26.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528361 2024-03-29T06:17:26.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:26.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528361 2024-03-29T06:17:26.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-03-29T06:17:26.644 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 193273528361 2024-03-29T06:17:26.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:17:27.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528361 -lt 193273528361 2024-03-29T06:17:27.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:27.006 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-373662154754 2024-03-29T06:17:27.006 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:27.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:17:27.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:27.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-373662154754 2024-03-29T06:17:27.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154754 2024-03-29T06:17:27.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 373662154754' 2024-03-29T06:17:27.011 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 373662154754 2024-03-29T06:17:27.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:17:27.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154755 -lt 373662154754 2024-03-29T06:17:27.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:17:27.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:27.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:27.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:17:27.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:17:27.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:17:27.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:17:27.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:17:27.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:17:27.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:17:27.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:17:28.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:17:28.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:17:28.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:28.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:28.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:17:28.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:17:28.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:17:28.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:17:28.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:17:28.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-29T06:17:28.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:17:28.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:17:28.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-29T06:17:28.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-29T06:17:28.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:17:28.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:17:28.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 222470"' 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 222470' 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 222471"' 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 222471' 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 222470 222471' 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:17:28.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:17:28.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 222470 2024-03-29T06:17:28.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:17:28.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/222473: /' 2024-03-29T06:17:28.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:17:28.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/222475: /' 2024-03-29T06:17:33.432 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.433 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:17:33.433 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:17:33.433 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:17:33.433 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:17:33.433 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.433 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:17:33.433 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:17:33.434 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:17:33.434 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:17:33.434 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:17:33.434 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:17:33.434 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:17:33.434 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:17:33.434 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:17:33.435 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:17:33.436 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:17:33.436 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:17:33.436 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:17:33.436 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:17:33.438 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:17:33.438 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:17:33.438 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:17:33.438 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:17:33.438 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:17:33.439 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:17:33.440 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:17:33.440 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:17:33.440 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:17:33.440 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:17:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:17:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:17:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:17:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:17:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:222475: start osd.9 2024-03-29T06:17:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:17:33.509 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:17:33.510 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:17:33.511 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:17:33.512 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:17:33.512 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:17:33.513 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:17:33.514 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:17:33.515 INFO:tasks.workunit.client.0.smithi123.stderr:222473: start osd.5 2024-03-29T06:17:33.515 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 2024-03-29T06:17:33.542+0000 7f8db19e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 2024-03-29T06:17:33.542+0000 7f8db19e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 2024-03-29T06:17:33.546+0000 7f8db19e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:17:38.276 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 0 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 2024-03-29T06:17:34.086+0000 7f8db19e87c0 -1 Falling back to public interface 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 1 2024-03-29T06:17:38.277 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 2 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 2024-03-29T06:17:37.518+0000 7f8db19e87c0 -1 osd.5 89 log_to_monitors true 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 3 2024-03-29T06:17:38.278 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.279 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:17:38.279 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:17:38.279 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:17:38.279 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:17:38.279 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 2024-03-29T06:17:33.470+0000 7ff3a51cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 2024-03-29T06:17:33.514+0000 7ff3a51cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 2024-03-29T06:17:33.514+0000 7ff3a51cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 0 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:38.280 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 2024-03-29T06:17:34.058+0000 7ff3a51cd7c0 -1 Falling back to public interface 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 1 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 2 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:38.281 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 2024-03-29T06:17:37.634+0000 7ff3a51cd7c0 -1 osd.9 89 log_to_monitors true 2024-03-29T06:17:38.282 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:38.282 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:38.282 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:17:38.282 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 3 2024-03-29T06:17:38.282 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:38.282 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:41.603 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:41.603 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:41.603 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:41.603 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:17:41.603 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 4 2024-03-29T06:17:41.603 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:41.603 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 90 last_clean_interval [87,89) [v2:127.0.0.1:6842/2093438403,v1:127.0.0.1:6843/2093438403] [v2:127.0.0.1:6844/2093438403,v1:127.0.0.1:6845/2093438403] exists,up 33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:17:41.604 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 1 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 2 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 3 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 4 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 5 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 6 2024-03-29T06:17:41.605 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 7 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 8 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: 9' 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836595 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836595 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595' 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /s.sh:985: wait_for_osd: sleep 1 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:17:41.606 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 4 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: osd.5 up in weight 1 up_from 95 up_thru 95 down_at 90 last_clean_interval [81,89) [v2:127.0.0.1:6874/1831437999,v1:127.0.0.1:6875/1831437999] [v2:127.0.0.1:6876/1831437999,v1:127.0.0.1:6877/1831437999] exists,up 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:17:41.607 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 1 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 2 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 3 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 4 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 5 2024-03-29T06:17:41.608 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 6 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 7 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 8 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: 9' 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836596 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836596 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596' 2024-03-29T06:17:41.609 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673069 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673069 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069' 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509535 2024-03-29T06:17:43.201 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509535 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535' 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710741 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710741 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535 3-300647710741' 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182477 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182477 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535 3-300647710741 4-107374182477' 2024-03-29T06:17:43.202 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535 3-300647710741 4-107374182477 5-408021893122' 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855427 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855427 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535 3-300647710741 4-107374182477 5-408021893122 6-150323855427' 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.203 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:17:43.206 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673070 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673070 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070' 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534' 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710742 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710742 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534 3-300647710742' 2024-03-29T06:17:43.207 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182478 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182478 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534 3-300647710742 4-107374182478' 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534 3-300647710742 4-107374182478 5-408021893123' 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855428 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855428 2024-03-29T06:17:43.208 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534 3-300647710742 4-107374182478 5-408021893123 6-150323855428' 2024-03-29T06:17:43.209 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.209 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:17:43.789 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-03-29T06:17:43.789 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-03-29T06:17:43.789 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534 3-300647710742 4-107374182478 5-408021893123 6-150323855428 7-171798691894' 2024-03-29T06:17:43.789 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528367 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528367 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534 3-300647710742 4-107374182478 5-408021893123 6-150323855428 7-171798691894 8-193273528367' 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836596 1-42949673070 2-64424509534 3-300647710742 4-107374182478 5-408021893123 6-150323855428 7-171798691894 8-193273528367 9-412316860418' 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836596 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:43.790 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836596 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836596 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.0 seq 21474836596 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836597 -lt 21474836596 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673070 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673070 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673070 2024-03-29T06:17:43.791 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.1 seq 42949673070 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_pg_stats: seq=171798691893 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535 3-300647710741 4-107374182477 5-408021893122 6-150323855427 7-171798691893' 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528366 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528366 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535 3-300647710741 4-107374182477 5-408021893122 6-150323855427 7-171798691893 8-193273528366' 2024-03-29T06:17:43.792 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509535 3-300647710741 4-107374182477 5-408021893122 6-150323855427 7-171798691893 8-193273528366 9-412316860419' 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836595 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836595 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836595 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836595' 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.0 seq 21474836595 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:17:43.793 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836597 -lt 21474836595 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673069 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673069 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673069 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-03-29T06:17:43.794 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.1 seq 42949673069 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673071 -lt 42949673069 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509535 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509535 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509535 2024-03-29T06:17:45.042 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.2 seq 64424509535 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509535 -lt 64424509535 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710741 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710741 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710741 2024-03-29T06:17:45.043 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710741' 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.3 seq 300647710741 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710742 -lt 300647710741 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182477 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182477 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182477 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.4 seq 107374182477 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:17:45.044 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182478 -lt 107374182477 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-408021893122 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cestats: ceph osd last-stat-seq 1 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673071 -lt 42949673070 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.045 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.2 seq 64424509534 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509535 -lt 64424509534 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710742 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710742 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.046 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710742 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710742' 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.3 seq 300647710742 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710742 -lt 300647710742 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182478 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182478 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182478 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-03-29T06:17:45.047 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.4 seq 107374182478 2024-03-29T06:17:45.048 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:17:45.048 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182478 -lt 107374182478 2024-03-29T06:17:45.048 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.048 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-408021893123 2024-03-29T06:17:45.964 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.964 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:17:45.964 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-408021893123 2024-03-29T06:17:45.964 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.964 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-03-29T06:17:45.964 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2024-03-29T06:17:45.964 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.5 seq 408021893123 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893123 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855428 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855428 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855428 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.6 seq 150323855428 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:17:45.965 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855428 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.7 seq 171798691894 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691894 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528367 2024-03-29T06:17:45.966 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:17:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528367 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-408021893122 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 408021893122' 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.5 seq 408021893122 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:17:46.133 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893122 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855427 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855427 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855427 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.6 seq 150323855427 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855427 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-03-29T06:17:46.134 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.7 seq 171798691893 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691893 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528366 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:17:46.135 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528366 2024-03-29T06:17:47.717 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: ccut -d - -f 2 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528367 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.8 seq 193273528367 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528367 -lt 193273528367 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860418 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860418 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-03-29T06:17:47.718 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: waiting osd.9 seq 412316860418 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860418 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:17:47.719 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:17:47.720 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:17:47.720 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:17:47.720 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:47.720 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:47.720 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T06:17:47.720 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:17:47.720 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:17:47.818 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephut -d - -f 2 2024-03-29T06:17:47.818 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528366 2024-03-29T06:17:47.818 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-03-29T06:17:47.818 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.8 seq 193273528366 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528367 -lt 193273528366 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860419 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860419 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860419' 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: waiting osd.9 seq 412316860419 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:17:47.819 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860419 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:17:47.820 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:47.821 INFO:tasks.workunit.client.0.smithi123.stderr:222475: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:47.821 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:17:47.821 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:17:47.821 INFO:tasks.workunit.client.0.smithi123.stderr:222475: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:17:47.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:17:48.464 INFO:tasks.workunit.client.0.smithi123.stderr:test/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:17:48.465 INFO:tasks.workunit.client.0.smithi123.stderr:222473: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:17:48.466 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:17:48.466 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:17:48.466 INFO:tasks.workunit.client.0.smithi123.stderr:222473: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:17:48.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:17:48.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:17:48.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 222471 2024-03-29T06:17:48.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:17:48.467 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:17:48.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:17:48.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:17:48.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:17:48.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:17:48.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:17:48.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:17:48.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:17:48.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:17:48.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:17:48.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:17:48.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:17:48.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:17:48.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:48.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:48.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:48.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:49.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:17:08.920172+0000 2024-03-29T06:17:49.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:17:49.349 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:17:49.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:17:08.920172+0000 2024-03-29T06:17:49.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:17:49.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:17:08.920172+0000 2024-03-29T06:17:49.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:17:49.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:17:49.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:17:49.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:17:49.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:49.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:49.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:49.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:49.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:08.920172+0000 '>' 2024-03-29T06:17:08.920172+0000 2024-03-29T06:17:49.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:17:50.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:17:50.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:17:50.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:17:50.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:50.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:50.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:50.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:50.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:08.920172+0000 '>' 2024-03-29T06:17:08.920172+0000 2024-03-29T06:17:50.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:17:51.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:17:51.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:17:51.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:17:51.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:17:51.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:17:51.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:17:51.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:50.197074+0000 '>' 2024-03-29T06:17:08.920172+0000 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 225925"' 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 225925' 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:17:52.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 225926"' 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 225926' 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 225925 225926' 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 225925 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/225929: /' 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:17:52.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/225930: /' 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:17:56.612 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: _ 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: hinfo_key 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: snapset 2024-03-29T06:17:56.613 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:17:56.614 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.614 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:17:56.614 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:17:56.614 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:17:56.614 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:17:56.614 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:17:56.614 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:17:56.617 INFO:tasks.workunit.client.0.smithi123.stderr:225929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:17:56.618 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:17:56.619 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:17:56.619 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:17:56.619 INFO:tasks.workunit.client.0.smithi123.stderr:225929: start osd.9 2024-03-29T06:17:56.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.821 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:17:56.821 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:17:56.821 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:17:56.821 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:17:56.821 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.821 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:17:56.822 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: _ 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: hinfo_key 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: snapset 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:17:56.823 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:17:56.824 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:17:56.824 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:17:56.824 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:17:56.825 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:17:56.826 INFO:tasks.workunit.client.0.smithi123.stderr:225930: start osd.5 2024-03-29T06:18:01.324 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 2024-03-29T06:17:56.649+0000 7efe875d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 2024-03-29T06:17:56.653+0000 7efe875d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 2024-03-29T06:17:56.653+0000 7efe875d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:18:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 0 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 2024-03-29T06:17:57.229+0000 7efe875d37c0 -1 Falling back to public interface 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:18:01.326 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 1 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 2 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 2024-03-29T06:18:00.401+0000 7efe875d37c0 -1 osd.9 98 log_to_monitors true 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.327 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:18:01.328 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 3 2024-03-29T06:18:01.328 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.328 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:18:01.719 INFO:tasks.workunit.client.0.smithi123.stderr:225929: osd.9 up in weight 1 up_from 103 up_thru 0 down_aovery_ops 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 2024-03-29T06:17:56.849+0000 7f59393757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 2024-03-29T06:17:56.853+0000 7f59393757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 2024-03-29T06:17:56.853+0000 7f59393757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:18:01.720 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 0 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 2024-03-29T06:17:57.393+0000 7f59393757c0 -1 Falling back to public interface 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 1 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:01.721 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 2 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 2024-03-29T06:18:00.789+0000 7f59393757c0 -1 osd.5 98 log_to_monitors true 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 3 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:01.722 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ct 99 last_clean_interval [96,98) [v2:127.0.0.1:6842/3458219407,v1:127.0.0.1:6843/3458219407] [v2:127.0.0.1:6844/3458219407,v1:127.0.0.1:6845/3458219407] exists,up 33487da7-fb1a-4701-ab10-066584f8d7a4 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:18:02.744 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 1 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 2 2024-03-29T06:18:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 3 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 4 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 5 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 6 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 7 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 8 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: 9' 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836602 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836602 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602' 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:02.746 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:18:02.747 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673075 2024-03-29T06:18:02.747 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673075 2024-03-29T06:18:02.747 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075' 2024-03-29T06:18:02.747 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.eph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 4 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: osd.5 up in weight 1 up_from 104 up_thru 95 down_at 99 last_clean_interval [95,98) [v2:127.0.0.1:6874/3295165350,v1:127.0.0.1:6875/3295165350] [v2:127.0.0.1:6876/3295165350,v1:127.0.0.1:6877/3295165350] exists,up 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:18:04.346 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:18:04.347 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 1 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 2 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 3 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 4 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 5 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 6 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 7 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 8 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: 9' 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:18:04.348 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.349 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:18:04.349 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836603 2024-03-29T06:18:04.349 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836603 2024-03-29T06:18:04.349 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603' 2024-03-29T06:18:04.711 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $iclient.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:18:04.711 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509540 2024-03-29T06:18:04.711 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509540 2024-03-29T06:18:04.711 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540' 2024-03-29T06:18:04.711 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710747 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710747 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540 3-300647710747' 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182483 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182483 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540 3-300647710747 4-107374182483' 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-03-29T06:18:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540 3-300647710747 4-107374182483 5-446676598786' 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855434 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855434 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540 3-300647710747 4-107374182483 5-446676598786 6-150323855434' 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691899 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540 3-300647710747 4-107374182483 5-446676598786 6-150323855434 7-171798691899' 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528372 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528372 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540 3-300647710747 4-107374182483 5-446676598786 6-150323855434 7-171798691899 8-193273528372' 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673075 2-64424509540 3-300647710747 4-107374182483 5-446676598786 6-150323855434 7-171798691899 8-193273528372 9-442381631490' 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836602 2024-03-29T06:18:05.821 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836602 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836602 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.0 seq 21474836602 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836602 -lt 21474836602 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673075 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673075 2024-03-29T06:18:05.822 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673075 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.1 seq 42949673075 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673076 -lt 42949673075 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509540 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:05.823 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:18:06.016 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtds 2024-03-29T06:18:06.016 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:18:06.016 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673077 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673077 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077' 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509542 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509542 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542' 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710748 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710748 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542 3-300647710748' 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.017 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182485 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182485 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542 3-300647710748 4-107374182485' 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542 3-300647710748 4-107374182485 5-446676598787' 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855435 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855435 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542 3-300647710748 4-107374182485 5-446676598787 6-150323855435' 2024-03-29T06:18:06.018 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.019 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691901 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691901 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542 3-300647710748 4-107374182485 5-446676598787 6-150323855435 7-171798691901' 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528373 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528373 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542 3-300647710748 4-107374182485 5-446676598787 6-150323855435 7-171798691901 8-193273528373' 2024-03-29T06:18:06.690 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631492 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631492 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673077 2-64424509542 3-300647710748 4-107374182485 5-446676598787 6-150323855435 7-171798691901 8-193273528373 9-442381631492' 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836603 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836603 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836603 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.0 seq 21474836603 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:18:06.691 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836603 -lt 21474836603 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673077 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673077 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673077 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673077' 2024-03-29T06:18:06.692 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.1 seq 42949673077 2024-03-29T06:18:06.762 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214est/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509540 2024-03-29T06:18:06.762 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:06.762 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509540 2024-03-29T06:18:06.762 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-03-29T06:18:06.762 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.2 seq 64424509540 2024-03-29T06:18:06.762 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509542 -lt 64424509540 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710747 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710747 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710747 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710747' 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.3 seq 300647710747 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710749 -lt 300647710747 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182483 2024-03-29T06:18:06.763 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182483 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182483 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.4 seq 107374182483 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182485 -lt 107374182483 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598786 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598786 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:06.764 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-03-29T06:18:07.949 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: : flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:18:07.949 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673077 -lt 42949673077 2024-03-29T06:18:07.949 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509542 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509542 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509542 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.2 seq 64424509542 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509542 -lt 64424509542 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-300647710748 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:07.950 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-300647710748 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710748 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 300647710748' 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.3 seq 300647710748 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710749 -lt 300647710748 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182485 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182485 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182485 2024-03-29T06:18:07.951 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182485' 2024-03-29T06:18:07.952 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.4 seq 107374182485 2024-03-29T06:18:07.952 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:18:07.952 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182485 -lt 107374182485 2024-03-29T06:18:07.952 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:07.952 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598787 2024-03-29T06:18:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //hoecho 'waiting osd.5 seq 446676598786' 2024-03-29T06:18:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.5 seq 446676598786 2024-03-29T06:18:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:18:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598786 -lt 446676598786 2024-03-29T06:18:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855434 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855434 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855434 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855434' 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.6 seq 150323855434 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855434 -lt 150323855434 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:18:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.7 seq 171798691899 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691899 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528372 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528372 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528372 2024-03-29T06:18:08.126 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-03-29T06:18:08.127 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.8 seq 193273528372 2024-03-29T06:18:08.127 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:18:08.127 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528373 -lt 193273528372 2024-03-29T06:18:08.945 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221me/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:08.945 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598787 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 446676598787' 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.5 seq 446676598787 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598788 -lt 446676598787 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855435 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855435 2024-03-29T06:18:08.946 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855435 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855435' 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.6 seq 150323855435 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855435 -lt 150323855435 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691901 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691901 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691901 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.7 seq 171798691901 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:18:08.947 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691901 2024-03-29T06:18:08.948 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:08.948 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528373 2024-03-29T06:18:08.948 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:08.948 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:18:08.948 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528373 2024-03-29T06:18:09.596 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_0: flush_pg_stats: for s in $seqs 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631490 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631490 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631490' 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: waiting osd.9 seq 442381631490 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631490 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:09.597 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:18:09.598 INFO:tasks.workunit.client.0.smithi123.stderr:225929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:18:09.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:pg_stats: cut -d - -f 2 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528373 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528373' 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.8 seq 193273528373 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528373 -lt 193273528373 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631492 2024-03-29T06:18:10.730 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631492 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631492 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631492' 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: waiting osd.9 seq 442381631492 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631492 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:18:10.731 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:225930: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:18:10.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 225926 2024-03-29T06:18:10.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:18:10.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:18:10.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:18:10.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:18:10.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:18:10.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:18:10.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:18:10.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229182"' 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229182' 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:18:10.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229184"' 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229184' 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 229182 229184' 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229182 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229185: /' 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:18:10.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229187: /' 2024-03-29T06:18:14.386 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.386 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:18:14.386 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:18:14.387 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:18:14.388 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:18:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:18:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:18:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:18:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:18:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:18:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:18:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:18:14.393 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:18:14.394 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:18:14.394 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:18:14.394 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:18:14.394 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:18:14.394 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:18:14.394 INFO:tasks.workunit.client.0.smithi123.stderr:229187: start osd.5 2024-03-29T06:18:14.394 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:18:14.671 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.671 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:18:14.671 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:18:14.671 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:18:14.671 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:18:14.671 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.671 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:18:14.672 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:18:14.673 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:18:14.675 INFO:tasks.workunit.client.0.smithi123.stderr:229185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:18:14.676 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:18:14.677 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:18:14.677 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:18:14.677 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:18:14.677 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:18:14.677 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:18:14.677 INFO:tasks.workunit.client.0.smithi123.stderr:229185: start osd.3 2024-03-29T06:18:14.678 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:18:19.121 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 2024-03-29T06:18:14.421+0000 7f9e285927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 2024-03-29T06:18:14.425+0000 7f9e285927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 2024-03-29T06:18:14.425+0000 7f9e285927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:18:19.122 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 0 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 2024-03-29T06:18:14.961+0000 7f9e285927c0 -1 Falling back to public interface 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 1 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 2 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 3 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 2024-03-29T06:18:18.689+0000 7f9e285927c0 -1 osd.5 106 log_to_monitors true 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.124 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:19.295 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpe229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 2024-03-29T06:18:14.705+0000 7f6d7e3d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 2024-03-29T06:18:14.709+0000 7f6d7e3d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 2024-03-29T06:18:14.709+0000 7f6d7e3d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.296 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 0 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 2024-03-29T06:18:15.269+0000 7f6d7e3d87c0 -1 Falling back to public interface 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 1 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:18:19.297 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 2 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 2024-03-29T06:18:18.345+0000 7f6d7e3d87c0 -1 osd.3 106 log_to_monitors true 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 3 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:19.298 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:21.471 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:21.471 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:21.471 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:21.471 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 4 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: osd.5 up in weight 1 up_from 111 up_thru 111 down_at 107 last_clean_interval [104,106) [v2:127.0.0.1:6826/2605025241,v1:127.0.0.1:6827/2605025241] [v2:127.0.0.1:6828/2605025241,v1:127.0.0.1:6829/2605025241] exists,up 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:18:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 1 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 2 2024-03-29T06:18:21.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 3 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 4 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 5 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 6 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 7 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 8 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: 9' 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:18:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:18:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836607 2024-03-29T06:18:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836607 2024-03-29T06:18:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607' 2024-03-29T06:18:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:21.630 INFO:tasks.workunit.client.0.smithi123.stderr:22rs.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:21.630 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:21.630 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:18:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 4 2024-03-29T06:18:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:229185: osd.3 up in weight 1 up_from 111 up_thru 111 down_at 107 last_clean_interval [70,106) [v2:127.0.0.1:6874/2432895959,v1:127.0.0.1:6875/2432895959] [v2:127.0.0.1:6876/2432895959,v1:127.0.0.1:6877/2432895959] exists,up 75727be0-e71a-4f24-99d4-cdaee9b3727e 2024-03-29T06:18:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:18:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:18:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:18:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:18:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:18:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:18:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:18:21.633 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:18:21.633 INFO:tasks.workunit.client.0.smithi123.stderr:229185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:18:21.633 INFO:tasks.workunit.client.0.smithi123.stderr:229185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:18:21.633 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:18:21.633 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:18:21.633 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:18:21.634 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:18:21.634 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:18:21.635 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:18:21.635 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:18:21.635 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:18:21.635 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:18:21.635 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:18:21.635 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 1 2024-03-29T06:18:21.636 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 2 2024-03-29T06:18:21.636 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 3 2024-03-29T06:18:21.636 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 4 2024-03-29T06:18:21.636 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 5 2024-03-29T06:18:21.636 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 6 2024-03-29T06:18:21.636 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 7 2024-03-29T06:18:21.637 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 8 2024-03-29T06:18:21.637 INFO:tasks.workunit.client.0.smithi123.stderr:229185: 9' 2024-03-29T06:18:21.637 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:18:21.637 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:21.637 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:18:21.637 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836608 2024-03-29T06:18:21.638 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836608 2024-03-29T06:18:21.638 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608' 2024-03-29T06:18:21.638 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:22.907 INFO:tasks.workunit.client.0.smithi123.stderr:2299187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:18:22.907 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673081 2024-03-29T06:18:22.907 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673081 2024-03-29T06:18:22.907 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081' 2024-03-29T06:18:22.907 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509546 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509546 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546' 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546 3-476741369858' 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182489 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182489 2024-03-29T06:18:22.908 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546 3-476741369858 4-107374182489' 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546 3-476741369858 4-107374182489 5-476741369858' 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855439 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855439 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546 3-476741369858 4-107374182489 5-476741369858 6-150323855439' 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:22.909 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:18:23.048 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202:185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:18:23.048 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673082 2024-03-29T06:18:23.048 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673082 2024-03-29T06:18:23.048 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082' 2024-03-29T06:18:23.048 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509547 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509547 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547' 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547 3-476741369859' 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182490 2024-03-29T06:18:23.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182490 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547 3-476741369859 4-107374182490' 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547 3-476741369859 4-107374182490 5-476741369859' 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855440 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855440 2024-03-29T06:18:23.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547 3-476741369859 4-107374182490 5-476741369859 6-150323855440' 2024-03-29T06:18:23.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:18:23.643 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691906 2024-03-29T06:18:23.643 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691906 2024-03-29T06:18:23.644 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547 3-476741369859 4-107374182490 5-476741369859 6-150323855440 7-171798691906' 2024-03-29T06:18:23.644 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.644 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:18:23.644 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528379 2024-03-29T06:18:23.644 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528379 2024-03-29T06:18:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547 3-476741369859 4-107374182490 5-476741369859 6-150323855440 7-171798691906 8-193273528379' 2024-03-29T06:18:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:18:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631497 2024-03-29T06:18:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631497 2024-03-29T06:18:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836608 1-42949673082 2-64424509547 3-476741369859 4-107374182490 5-476741369859 6-150323855440 7-171798691906 8-193273528379 9-442381631497' 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836608 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836608 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836608 2024-03-29T06:18:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836608' 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.0 seq 21474836608 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836609 -lt 21474836608 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673082 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:18:23.647 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673082 2024-03-29T06:18:23.648 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:23.648 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673082 2024-03-29T06:18:23.648 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673082' 2024-03-29T06:18:23.649 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.1 seq 42949673082 2024-03-29T06:18:24.808 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flus flush_pg_stats: seq=171798691905 2024-03-29T06:18:24.808 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691905 2024-03-29T06:18:24.808 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546 3-476741369858 4-107374182489 5-476741369858 6-150323855439 7-171798691905' 2024-03-29T06:18:24.808 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:24.808 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:18:24.808 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528378 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528378 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546 3-476741369858 4-107374182489 5-476741369858 6-150323855439 7-171798691905 8-193273528378' 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631496 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631496 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509546 3-476741369858 4-107374182489 5-476741369858 6-150323855439 7-171798691905 8-193273528378 9-442381631496' 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836607 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:18:24.809 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836607 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836607 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.0 seq 21474836607 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836606 -lt 21474836607 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:18:24.810 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836609 -lt 21474836607 2024-03-29T06:18:24.811 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:24.811 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:24.811 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673081 2024-03-29T06:18:24.811 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:18:24.811 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673081 2024-03-29T06:18:25.049 INFO:tasks.workunit.client.0.smithi123.stderr:229187: h_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:18:25.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673083 -lt 42949673082 2024-03-29T06:18:25.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:25.049 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509547 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509547 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509547 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509547' 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.2 seq 64424509547 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509547 -lt 64424509547 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369859 2024-03-29T06:18:25.050 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369859 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369859' 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.3 seq 476741369859 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369859 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182490 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:25.051 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182490 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182490 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.4 seq 107374182490 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182490 -lt 107374182490 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:25.052 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369859 2024-03-29T06:18:26.103 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubu//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:26.103 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673081 2024-03-29T06:18:26.103 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673081' 2024-03-29T06:18:26.103 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.1 seq 42949673081 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673083 -lt 42949673081 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509546 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509546 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509546 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509546' 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.2 seq 64424509546 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509548 -lt 64424509546 2024-03-29T06:18:26.104 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369858 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369858 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369858' 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.3 seq 476741369858 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369860 -lt 476741369858 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182489 2024-03-29T06:18:26.105 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:26.106 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:18:26.106 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182489 2024-03-29T06:18:26.106 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:26.106 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182489 2024-03-29T06:18:26.106 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182489' 2024-03-29T06:18:26.106 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.4 seq 107374182489 2024-03-29T06:18:26.349 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtesntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369859 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 476741369859' 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.5 seq 476741369859 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369860 -lt 476741369859 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855440 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:18:26.350 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855440 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855440 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855440' 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.6 seq 150323855440 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855441 -lt 150323855440 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691906 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691906 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691906 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-03-29T06:18:26.351 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.7 seq 171798691906 2024-03-29T06:18:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:18:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691907 -lt 171798691906 2024-03-29T06:18:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528379 2024-03-29T06:18:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:18:26.352 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528379 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stat/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182491 -lt 107374182489 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369858 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369858 2024-03-29T06:18:27.472 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 476741369858' 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.5 seq 476741369858 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369860 -lt 476741369858 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855439 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855439 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855439 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855439' 2024-03-29T06:18:27.473 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.6 seq 150323855439 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855441 -lt 150323855439 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691905 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691905 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691905 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691905' 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.7 seq 171798691905 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691907 -lt 171798691905 2024-03-29T06:18:27.474 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperts: cut -d - -f 2 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528379 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.8 seq 193273528379 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528379 -lt 193273528379 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631497 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631497 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:28.092 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631497 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631497' 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: waiting osd.9 seq 442381631497 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631497 -lt 442381631497 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:18:28.093 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:28.094 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:18:28.095 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:18:28.095 INFO:tasks.workunit.client.0.smithi123.stderr:229185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:18:28.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:18:28.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:18:28.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229184 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:s.sh:2211: flush_pg_stats: echo 8-193273528378 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528378 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528378 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-03-29T06:18:28.739 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.8 seq 193273528378 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528379 -lt 193273528378 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631496 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631496 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631496 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631496' 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: waiting osd.9 seq 442381631496 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631498 -lt 442381631496 2024-03-29T06:18:28.740 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:18:28.741 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:18:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:229187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:18:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:18:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:229187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:18:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:18:29.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:18:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:18:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:18:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:18:29.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:18:29.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:18:29.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:18:29.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:18:29.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:18:29.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:29.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:18:30.024 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:18:30.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:30.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:18:30.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:30.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:18:30.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:18:30.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:18:30.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:18:30.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:18:30.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:18:30.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:18:30.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:18:30.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:50.197074+0000 '>' 2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:30.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:18:31.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:18:31.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:18:31.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:18:31.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:18:31.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:18:31.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:18:31.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:18:31.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:50.197074+0000 '>' 2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:31.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:18:32.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:18:32.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:18:32.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:18:32.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:18:32.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:18:32.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:18:32.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:18:32.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:50.197074+0000 '>' 2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:32.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:18:33.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:18:33.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:18:33.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:18:33.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:18:33.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:18:33.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:18:33.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:18:34.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:17:50.197074+0000 '>' 2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:34.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:18:35.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:18:35.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:18:35.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:18:35.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:18:35.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:18:35.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:18:35.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:18:30.195582+0000 '>' 2024-03-29T06:17:50.197074+0000 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232659"' 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232659' 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:18:35.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232660"' 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232660' 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 232659 232660' 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232659 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232662: /' 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:18:35.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232664: /' 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:18:39.340 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:18:39.341 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: _ 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: hinfo_key 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: snapset 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:18:39.342 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:18:39.344 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:18:39.344 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:18:39.344 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:18:39.344 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:18:39.344 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:18:39.344 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:18:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:18:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:232662: start osd.3 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:18:39.926 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: _ 2024-03-29T06:18:39.927 INFO:tasks.workunit.client.0.smithi123.stderr:232664: hinfo_key 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: snapset 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:18:39.928 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:18:39.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:18:39.931 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:18:39.932 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:18:39.932 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:18:39.932 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:18:39.932 INFO:tasks.workunit.client.0.smithi123.stderr:232664: start osd.5 2024-03-29T06:18:43.655 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 2024-03-29T06:18:39.376+0000 7fe5f88237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 2024-03-29T06:18:39.380+0000 7fe5f88237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 2024-03-29T06:18:39.380+0000 7fe5f88237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:18:43.656 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 0 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 2024-03-29T06:18:39.932+0000 7fe5f88237c0 -1 Falling back to public interface 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 1 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:43.657 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 2 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 2024-03-29T06:18:43.136+0000 7fe5f88237c0 -1 osd.3 112 log_to_monitors true 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 3 2024-03-29T06:18:43.658 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 2024-03-29T06:18:39.956+0000 7f18c40ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 2024-03-29T06:18:39.968+0000 7f18c40ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 2024-03-29T06:18:39.972+0000 7f18c40ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:18:44.700 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 0 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 2024-03-29T06:18:40.544+0000 7f18c40ab7c0 -1 Falling back to public interface 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:18:44.701 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 1 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 2 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 2024-03-29T06:18:44.004+0000 7f18c40ab7c0 -1 osd.5 112 log_to_monitors true 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:44.702 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:44.703 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:18:44.703 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 3 2024-03-29T06:18:44.703 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:44.703 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:45.581 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:18:45.581 INFO:tasks.workunit.client.0.smithi123.stderr:232662: osd.3 up in weight 1 up_from 116 up_thru 111 down_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6826/1088983516,v1:127.0.0.1:6827/1088983516] [v2:127.0.0.1:6828/1088983516,v1:127.0.0.1:6829/1088983516] exists,up 75727be0-e71a-4f24-99d4-cdaee9b3727e 2024-03-29T06:18:45.581 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:18:45.582 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 1 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 2 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 3 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 4 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 5 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 6 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 7 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 8 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: 9' 2024-03-29T06:18:45.583 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836614 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836614 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614' 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673088 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673088 2024-03-29T06:18:45.584 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088' 2024-03-29T06:18:47.297 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:18:47.297 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:18:47.297 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:18:47.297 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 4 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: osd.5 up in weight 1 up_from 118 up_thru 111 down_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6874/2677002298,v1:127.0.0.1:6875/2677002298] [v2:127.0.0.1:6876/2677002298,v1:127.0.0.1:6877/2677002298] exists,up 1c46e833-5128-4294-b759-7c4a6d7535b2 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:18:47.298 INFO:tasks.workunit.client.0.smithi123.stderr:232664: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 1 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 2 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 3 2024-03-29T06:18:47.299 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 4 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 5 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 6 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 7 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 8 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: 9' 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836616 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836616 2024-03-29T06:18:47.300 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616' 2024-03-29T06:18:47.626 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd /qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509553 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509553 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553' 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553 3-498216206338' 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182496 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182496 2024-03-29T06:18:47.627 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553 3-498216206338 4-107374182496' 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553 3-498216206338 4-107374182496 5-506806140930' 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855446 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855446 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553 3-498216206338 4-107374182496 5-506806140930 6-150323855446' 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691912 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691912 2024-03-29T06:18:47.628 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553 3-498216206338 4-107374182496 5-506806140930 6-150323855446 7-171798691912' 2024-03-29T06:18:47.629 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.910 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qin $ids 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673089 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673089 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089' 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509554 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509554 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554' 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206339 2024-03-29T06:18:48.911 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206339 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554 3-498216206339' 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182497 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182497 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554 3-498216206339 4-107374182497' 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554 3-498216206339 4-107374182497 5-506806140931' 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855447 2024-03-29T06:18:48.912 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855447 2024-03-29T06:18:48.913 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554 3-498216206339 4-107374182497 5-506806140931 6-150323855447' 2024-03-29T06:18:48.913 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.913 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:18:48.934 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpera/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528385 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528385 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553 3-498216206338 4-107374182496 5-506806140930 6-150323855446 7-171798691912 8-193273528385' 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631503 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631503 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836614 1-42949673088 2-64424509553 3-498216206338 4-107374182496 5-506806140930 6-150323855446 7-171798691912 8-193273528385 9-442381631503' 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836614 2024-03-29T06:18:48.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836614 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836614 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836614' 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.0 seq 21474836614 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836616 -lt 21474836614 2024-03-29T06:18:48.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673088 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673088 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673088 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673088' 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.1 seq 42949673088 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673088 -lt 42949673088 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509553 2024-03-29T06:18:48.937 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/ceps.sh:2202: flush_pg_stats: seq=171798691913 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691913 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554 3-498216206339 4-107374182497 5-506806140931 6-150323855447 7-171798691913' 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528386 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528386 2024-03-29T06:18:49.504 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554 3-498216206339 4-107374182497 5-506806140931 6-150323855447 7-171798691913 8-193273528386' 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631504 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631504 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673089 2-64424509554 3-498216206339 4-107374182497 5-506806140931 6-150323855447 7-171798691913 8-193273528386 9-442381631504' 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836616 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836616 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836616 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836616' 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.0 seq 21474836616 2024-03-29T06:18:49.505 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836616 -lt 21474836616 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673089 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673089 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673089 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673089' 2024-03-29T06:18:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.1 seq 42949673089 2024-03-29T06:18:49.930 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:18:49.931 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509553 2024-03-29T06:18:49.931 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:49.931 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509553 2024-03-29T06:18:49.931 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509553' 2024-03-29T06:18:49.931 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.2 seq 64424509553 2024-03-29T06:18:49.931 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:18:49.932 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509553 -lt 64424509553 2024-03-29T06:18:49.932 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:49.932 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206338 2024-03-29T06:18:49.932 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:49.932 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:18:49.932 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206338 2024-03-29T06:18:49.932 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:49.933 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-29T06:18:49.933 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206338' 2024-03-29T06:18:49.933 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.3 seq 498216206338 2024-03-29T06:18:49.933 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:18:49.933 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206340 -lt 498216206338 2024-03-29T06:18:49.933 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:49.934 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182496 2024-03-29T06:18:49.934 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:49.934 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:18:49.934 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182496 2024-03-29T06:18:49.934 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:49.934 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182496 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182496' 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.4 seq 107374182496 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182498 -lt 107374182496 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140930 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:18:49.935 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140930 2024-03-29T06:18:49.936 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:50.811 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5068:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673090 -lt 42949673089 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509554 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509554 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509554 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509554' 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.2 seq 64424509554 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:18:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509555 -lt 64424509554 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206339 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206339 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206339 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206339' 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.3 seq 498216206339 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206340 -lt 498216206339 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182497 2024-03-29T06:18:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182497 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182497 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182497' 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.4 seq 107374182497 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182498 -lt 107374182497 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:50.814 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140931 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232664:06140930 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 506806140930' 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.5 seq 506806140930 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140930 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855446 2024-03-29T06:18:51.246 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855446 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855446 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855446' 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.6 seq 150323855446 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855448 -lt 150323855446 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691912 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:18:51.247 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691912 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691912 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691912' 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.7 seq 171798691912 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691913 -lt 171798691912 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528385 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528385 2024-03-29T06:18:51.248 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:51.249 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528385 2024-03-29T06:18:51.249 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528385' 2024-03-29T06:18:51.249 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.8 seq 193273528385 2024-03-29T06:18:51.249 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:18:51.817 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:51.817 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:18:51.817 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140931 2024-03-29T06:18:51.817 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:51.817 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-03-29T06:18:51.818 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 506806140931' 2024-03-29T06:18:51.818 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.5 seq 506806140931 2024-03-29T06:18:51.818 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:18:51.818 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140931 2024-03-29T06:18:51.818 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855447 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855447 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855447 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855447' 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.6 seq 150323855447 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855448 -lt 150323855447 2024-03-29T06:18:51.819 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691913 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691913 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691913 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691913' 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.7 seq 171798691913 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691914 -lt 171798691913 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528386 2024-03-29T06:18:51.820 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:51.821 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:18:51.821 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528386 2024-03-29T06:18:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: f93273528386 -lt 193273528385 2024-03-29T06:18:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631503 2024-03-29T06:18:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631503 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631503 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631503' 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: waiting osd.9 seq 442381631503 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631505 -lt 442381631503 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:18:52.620 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:18:52.621 INFO:tasks.workunit.client.0.smithi123.stderr:232662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:18:52.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:18:52.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:18:52.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232660 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:lush_pg_stats: cut -d - -f 2 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528386 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528386' 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.8 seq 193273528386 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528386 -lt 193273528386 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:18:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631504 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631504 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631504 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631504' 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: waiting osd.9 seq 442381631504 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631505 -lt 442381631504 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:18:53.491 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:232664: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:18:53.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:18:53.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:18:53.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:18:53.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:18:53.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:18:53.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:18:53.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:18:53.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:18:53.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:18:53.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:18:53.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:18:53.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:18:53.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:18:53.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:18:53.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:18:53.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:18:53.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:18:53.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:18:53.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:18:53.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:18:53.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:18:53.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:18:53.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:18:53.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:18:53.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:18:53.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:18:53.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:18:53.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:18:53.740 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:53.740 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:53.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:18:53.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:18:53.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:18:53.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:18:53.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:18:53.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:18:53.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:18:53.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:18:53.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:18:53.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:18:53.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:18:53.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:18:53.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:18:53.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:18:53.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:18:53.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:18:53.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:18:53.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:18:53.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:18:53.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:18:53.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:53.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:53.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:18:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:18:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:18:53.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:18:53.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:18:53.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:53.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:53.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-29T06:18:53.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:18:53.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T06:18:54.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:18:54.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:18:54.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:18:54.156 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:18:54.156 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:54.156 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:54.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:54.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T06:18:54.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:18:54.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:18:54.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:18:54.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:18:54.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:18:54.369 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:18:54.369 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:18:54.369 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:18:54.369 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:18:54.370 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:18:54.370 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:54.370 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:54.371 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:18:54.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:18:54.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:18:54.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:18:54.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:18:54.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:18:54.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:18:54.479 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:18:54.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:18:54.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-29T06:18:54.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:18:54.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:18:54.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:18:54.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:18:54.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:18:54.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:18:55.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:18:55.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:18:55.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:18:55.450 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:18:55.450 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:55.450 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:55.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:55.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:18:55.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:18:55.487 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-29T06:18:55.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:18:55.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:18:55.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:18:55.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:18:55.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f17eef16-1f13-43e6-8b82-6a89f68b0964 2024-03-29T06:18:55.494 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 f17eef16-1f13-43e6-8b82-6a89f68b0964 2024-03-29T06:18:55.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f17eef16-1f13-43e6-8b82-6a89f68b0964' 2024-03-29T06:18:55.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:18:55.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPXQZmMAlgHhAAoXYePT6fU5ImAzLiRqw7Nw== 2024-03-29T06:18:55.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPXQZmMAlgHhAAoXYePT6fU5ImAzLiRqw7Nw=="}' 2024-03-29T06:18:55.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f17eef16-1f13-43e6-8b82-6a89f68b0964 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:18:55.856 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:18:55.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:18:55.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPXQZmMAlgHhAAoXYePT6fU5ImAzLiRqw7Nw== --osd-uuid f17eef16-1f13-43e6-8b82-6a89f68b0964 2024-03-29T06:18:55.912 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:18:55.908+0000 7fc1cd0697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:55.945 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:18:55.944+0000 7fc1cd0697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:55.946 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:18:55.944+0000 7fc1cd0697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:18:55.947 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:18:55.944+0000 7fc1cd0697c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:18:55.947 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:18:55.944+0000 7fc1cd0697c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:18:55.947 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:18:55.944+0000 7fc1cd0697c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:18:56.045 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:18:56.040+0000 7fc1cd0697c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:19:02.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:19:02.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:19:02.325 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:19:02.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:19:02.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:19:02.704 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:19:02.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:19:02.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:19:02.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:19:02.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:19:02.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:19:02.736 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:02.732+0000 7f4c3efbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:02.738 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:02.736+0000 7f4c3efbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:02.739 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:02.736+0000 7f4c3efbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:03.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:19:03.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:19:03.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:19:03.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:19:03.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:19:03.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:03.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:19:03.045 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:19:03.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:03.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:03.299 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:03.296+0000 7f4c3efbf7c0 -1 Falling back to public interface 2024-03-29T06:19:03.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:04.371 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:19:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:19:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:04.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:04.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:05.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:05.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:05.698 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:19:05.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:19:05.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:05.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:05.754 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:05.752+0000 7f4c3efbf7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:19:06.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:07.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:07.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:07.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:19:07.090 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:19:07.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:07.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:07.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:08.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:08.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:08.466 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:19:08.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:19:08.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:08.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:08.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:09.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:09.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:09.868 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:19:09.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:19:09.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:09.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:10.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:11.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:11.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:11.191 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:19:11.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:19:11.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:11.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:11.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:12.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:12.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:12.511 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:19:12.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:19:12.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:12.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:12.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:13.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:13.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:13.844 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:19:13.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:19:13.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:13.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:14.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:15.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:15.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:15.170 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:19:15.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:19:15.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:15.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:15.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:16.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:16.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:16.491 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:19:16.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:19:16.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:16.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:16.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:17.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:17.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:17.815 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:19:17.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:19:17.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:17.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:18.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:19.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:19.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:19.130 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:19:19.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:19:19.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:19.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:19.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:20.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:20.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:20.602 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:19:20.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:19:20.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:20.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:22.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:22.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:22.195 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:19:22.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:19:22.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:22.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:22.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:23.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:23.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:23.949 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:19:23.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:19:23.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:23.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:24.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:25.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:25.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:25.870 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:19:25.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:19:25.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:25.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:26.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:27.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:27.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:27.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:19:27.933 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:19:27.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:27.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:28.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:29.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:29.470 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:19:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:19:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:30.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:31.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:31.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:31.116 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:19:31.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:19:31.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:31.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:19:31.422 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1519472607,v1:127.0.0.1:6803/1519472607] [v2:127.0.0.1:6804/1519472607,v1:127.0.0.1:6805/1519472607] exists,up f17eef16-1f13-43e6-8b82-6a89f68b0964 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:19:31.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:19:31.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:19:31.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:19:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:19:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:19:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:19:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:19:31.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:19:31.429 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 3fd1092d-adf0-469f-9b64-7090c461c6a5 2024-03-29T06:19:31.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fd1092d-adf0-469f-9b64-7090c461c6a5 2024-03-29T06:19:31.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3fd1092d-adf0-469f-9b64-7090c461c6a5' 2024-03-29T06:19:31.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:19:31.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzXQZmV/rgGhAAFhNPxyw/Q9HmVH1MIXicsQ== 2024-03-29T06:19:31.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzXQZmV/rgGhAAFhNPxyw/Q9HmVH1MIXicsQ=="}' 2024-03-29T06:19:31.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fd1092d-adf0-469f-9b64-7090c461c6a5 -i td/osd-scrub-repair/1/new.json 2024-03-29T06:19:31.813 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:19:31.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:19:31.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzXQZmV/rgGhAAFhNPxyw/Q9HmVH1MIXicsQ== --osd-uuid 3fd1092d-adf0-469f-9b64-7090c461c6a5 2024-03-29T06:19:31.863 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:31.859+0000 7f2948d327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:31.898 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:31.895+0000 7f2948d327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:31.899 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:31.895+0000 7f2948d327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:31.900 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:31.895+0000 7f2948d327c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:19:31.900 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:31.895+0000 7f2948d327c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:19:31.900 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:31.895+0000 7f2948d327c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:19:31.982 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:31.979+0000 7f2948d327c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:19:38.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:19:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:19:38.968 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:19:38.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:19:38.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:19:39.345 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:19:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:19:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:19:39.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:19:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:19:39.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:19:39.376 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:39.371+0000 7f10d11647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:39.375+0000 7f10d11647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:39.375+0000 7f10d11647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:19:39.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:19:39.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:19:39.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:19:39.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:19:39.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:19:39.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:39.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:19:39.698 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:19:39.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:39.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:39.951 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:39.947+0000 7f10d11647c0 -1 Falling back to public interface 2024-03-29T06:19:40.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:41.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:41.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:41.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:19:41.015 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:19:41.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:41.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:41.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:42.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:42.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:42.330 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:19:42.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:19:42.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:42.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:42.618 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:19:42.595+0000 7f10d11647c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:19:42.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:43.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:43.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:43.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:19:43.632 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:19:43.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:43.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:43.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:44.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:44.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:44.991 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:19:44.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:19:44.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:44.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:45.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:46.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:46.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:46.407 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:19:46.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:19:46.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:46.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:46.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:47.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:47.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:47.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:19:47.955 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:19:47.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:47.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:48.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:49.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:49.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:49.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:19:49.353 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:19:49.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:49.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:49.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:50.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:50.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:50.765 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:19:50.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:19:50.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:50.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:51.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:52.105 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:19:52.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:19:52.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:52.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:52.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:53.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:53.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:53.425 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:19:53.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:19:53.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:53.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:53.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:54.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:54.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:54.760 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:19:54.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:19:54.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:54.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:55.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:56.183 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:19:56.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:19:56.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:56.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:56.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:57.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:57.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:57.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:19:57.501 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:19:57.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:57.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:58.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:19:59.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:19:59.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:19:59.264 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:19:59.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:19:59.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:19:59.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:19:59.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:00.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:00.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:00.904 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:20:00.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:20:00.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:00.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:20:01.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:02.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:02.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:02.477 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:20:02.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:20:02.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:02.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:20:02.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:03.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:03.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:03.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:20:03.824 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:20:03.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:03.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:20:04.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:05.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:05.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:05.389 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:20:05.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:20:05.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:05.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:20:06.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:07.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:07.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:07.165 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:20:07.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:20:07.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:07.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:20:07.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:08.697 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:20:08.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:20:08.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:08.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:20:09.063 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2084365397,v1:127.0.0.1:6811/2084365397] [v2:127.0.0.1:6812/2084365397,v1:127.0.0.1:6813/2084365397] exists,up 3fd1092d-adf0-469f-9b64-7090c461c6a5 2024-03-29T06:20:09.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:20:09.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:20:09.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:20:09.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:20:09.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-29T06:20:09.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:20:09.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:20:09.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:20:09.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:20:09.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:20:09.067 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:20:09.067 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:20:09.067 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:20:09.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:20:09.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:20:09.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:20:09.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:20:09.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:20:09.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:20:09.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:20:09.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78a30d7d-98e5-4214-992a-28ef9c330c46 2024-03-29T06:20:09.070 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 78a30d7d-98e5-4214-992a-28ef9c330c46 2024-03-29T06:20:09.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 78a30d7d-98e5-4214-992a-28ef9c330c46' 2024-03-29T06:20:09.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:20:09.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZXQZmMTRyBRAAVcR2V+qOetb5E1UpbFjx4Q== 2024-03-29T06:20:09.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZXQZmMTRyBRAAVcR2V+qOetb5E1UpbFjx4Q=="}' 2024-03-29T06:20:09.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78a30d7d-98e5-4214-992a-28ef9c330c46 -i td/osd-scrub-repair/2/new.json 2024-03-29T06:20:09.452 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:20:09.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T06:20:09.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZXQZmMTRyBRAAVcR2V+qOetb5E1UpbFjx4Q== --osd-uuid 78a30d7d-98e5-4214-992a-28ef9c330c46 2024-03-29T06:20:09.498 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:09.494+0000 7fb8e2b0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:09.553 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:09.550+0000 7fb8e2b0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:09.555 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:09.550+0000 7fb8e2b0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:09.555 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:09.550+0000 7fb8e2b0c7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:20:09.555 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:09.550+0000 7fb8e2b0c7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:20:09.555 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:09.550+0000 7fb8e2b0c7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:20:09.638 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:09.634+0000 7fb8e2b0c7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T06:20:16.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T06:20:16.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:20:16.549 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T06:20:16.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T06:20:16.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:20:16.940 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T06:20:16.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T06:20:16.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:20:16.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:20:16.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:20:16.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:20:16.972 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:16.970+0000 7f7912fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:16.973 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:16.974+0000 7f7912fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:16.975 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:16.974+0000 7f7912fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:17.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T06:20:17.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:20:17.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:20:17.285 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:20:17.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:20:17.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:20:17.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:17.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:20:17.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:17.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:17.532 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:17.530+0000 7f7912fd67c0 -1 Falling back to public interface 2024-03-29T06:20:17.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:18.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:18.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:18.623 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:20:18.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:20:18.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:18.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:18.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:19.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:19.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:19.944 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:20:19.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:20:19.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:19.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:20.194 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:20.194+0000 7f7912fd67c0 -1 osd.2 0 log_to_monitors true 2024-03-29T06:20:20.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:21.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:21.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:21.283 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:20:21.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:20:21.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:21.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:21.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:22.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:22.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:22.609 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:20:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:20:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:22.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:23.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:24.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:24.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:24.150 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:20:24.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:20:24.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:24.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:24.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:25.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:20:25.475 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:20:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:25.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:25.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:26.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:26.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:26.795 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:20:26.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:20:26.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:26.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:28.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:28.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:28.159 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:20:28.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:20:28.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:28.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:28.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:29.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:29.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:29.481 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:20:29.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:20:29.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:29.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:29.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:30.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:30.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:30.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:20:30.802 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:20:30.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:30.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:31.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:32.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:32.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:32.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:20:32.257 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:20:32.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:32.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:32.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:33.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:33.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:33.653 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:20:33.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:20:33.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:33.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:33.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:34.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:34.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:34.974 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:20:34.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:20:34.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:34.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:35.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:36.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:36.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:36.341 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:20:36.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:20:36.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:36.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:36.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:38.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:38.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:38.002 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:20:38.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:20:38.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:38.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:38.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:39.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:39.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:39.519 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:20:39.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:20:39.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:39.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:40.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:41.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:41.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:41.083 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:20:41.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:20:41.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:41.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:41.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:42.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:42.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:42.902 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:20:42.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:20:42.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:42.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:43.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:44.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:44.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:44.523 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:20:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:20:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:45.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:46.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:46.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:46.389 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:20:46.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:20:46.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:46.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:20:46.770 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2586883038,v1:127.0.0.1:6819/2586883038] [v2:127.0.0.1:6820/2586883038,v1:127.0.0.1:6821/2586883038] exists,up 78a30d7d-98e5-4214-992a-28ef9c330c46 2024-03-29T06:20:46.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:20:46.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:20:46.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:20:46.776 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:20:46.777 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:20:46.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:20:46.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:20:46.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:20:46.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:20:46.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:20:46.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:20:46.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:20:46.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9050fb11-c4b9-4740-bfb4-e3aba24dc02c 2024-03-29T06:20:46.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9050fb11-c4b9-4740-bfb4-e3aba24dc02c' 2024-03-29T06:20:46.779 INFO:tasks.workunit.client.0.smithi123.stdout:add osd3 9050fb11-c4b9-4740-bfb4-e3aba24dc02c 2024-03-29T06:20:46.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:20:46.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+XQZmXVeFLxAAIy/DEXXG9Hjs8WoA04x2fQ== 2024-03-29T06:20:46.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+XQZmXVeFLxAAIy/DEXXG9Hjs8WoA04x2fQ=="}' 2024-03-29T06:20:46.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9050fb11-c4b9-4740-bfb4-e3aba24dc02c -i td/osd-scrub-repair/3/new.json 2024-03-29T06:20:47.164 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:20:47.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-29T06:20:47.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+XQZmXVeFLxAAIy/DEXXG9Hjs8WoA04x2fQ== --osd-uuid 9050fb11-c4b9-4740-bfb4-e3aba24dc02c 2024-03-29T06:20:47.215 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:47.213+0000 7f11efc2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:47.266 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:47.265+0000 7f11efc2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:47.267 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:47.265+0000 7f11efc2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:47.268 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:47.265+0000 7f11efc2b7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:20:47.268 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:47.265+0000 7f11efc2b7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:20:47.268 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:47.265+0000 7f11efc2b7c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-29T06:20:47.349 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:47.349+0000 7f11efc2b7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-29T06:20:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-29T06:20:54.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:20:54.168 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd3 key to auth repository 2024-03-29T06:20:54.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-29T06:20:54.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:20:54.543 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.3 2024-03-29T06:20:54.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-29T06:20:54.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:20:54.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:20:54.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:20:54.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:20:54.575 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:54.573+0000 7fba1a6f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:54.589+0000 7fba1a6f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:54.592 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:54.589+0000 7fba1a6f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:20:54.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-29T06:20:54.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:20:54.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:20:54.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:20:54.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:20:54.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:54.867 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:20:54.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:20:54.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:54.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:20:55.151 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:55.149+0000 7fba1a6f97c0 -1 Falling back to public interface 2024-03-29T06:20:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:56.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:56.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:56.210 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:20:56.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:20:56.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:56.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:20:56.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:57.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:57.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:57.524 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:20:57.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:20:57.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:57.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:20:57.759 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:20:57.757+0000 7fba1a6f97c0 -1 osd.3 0 log_to_monitors true 2024-03-29T06:20:57.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:20:58.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:20:58.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:20:58.872 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:20:58.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:20:58.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:20:58.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:20:59.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:00.193 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:21:00.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:21:00.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:00.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:00.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:01.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:01.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:01.516 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:21:01.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:21:01.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:01.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:01.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:02.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:02.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:02.906 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:21:02.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:21:02.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:02.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:03.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:04.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:04.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:04.240 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:21:04.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:21:04.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:04.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:04.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:05.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:05.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:05.554 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:21:05.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:21:05.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:05.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:05.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:06.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:06.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:06.903 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:21:06.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:21:06.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:06.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:07.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:08.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:08.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:08.222 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:21:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:21:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:08.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:09.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:09.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:09.532 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:21:09.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:21:09.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:09.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:09.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:10.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:10.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:10.853 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:21:10.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:21:10.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:10.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:11.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:12.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:12.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:12.170 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:21:12.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:21:12.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:12.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:12.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:13.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:13.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:13.487 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:21:13.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:21:13.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:13.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:13.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:14.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:14.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:14.986 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:21:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:21:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:15.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:16.801 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:21:16.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:21:16.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:16.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:17.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:18.644 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:21:18.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:21:18.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:18.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:19.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:20.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:20.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:20.459 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:21:20.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:21:20.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:20.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:21.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:22.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:22.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:22.059 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:21:22.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:21:22.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:22.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:22.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:23.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:23.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:23.492 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:21:23.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:21:23.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:23.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:23.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:24.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:24.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:24.863 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:21:24.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:21:24.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:24.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:21:25.186 INFO:tasks.workunit.client.0.smithi123.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/207517685,v1:127.0.0.1:6827/207517685] [v2:127.0.0.1:6828/207517685,v1:127.0.0.1:6829/207517685] exists,up 9050fb11-c4b9-4740-bfb4-e3aba24dc02c 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-29T06:21:25.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:21:25.191 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:21:25.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-29T06:21:25.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:21:25.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6074cc0-f2e6-4844-a9c4-b03fb873edf9 2024-03-29T06:21:25.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a6074cc0-f2e6-4844-a9c4-b03fb873edf9' 2024-03-29T06:21:25.194 INFO:tasks.workunit.client.0.smithi123.stdout:add osd4 a6074cc0-f2e6-4844-a9c4-b03fb873edf9 2024-03-29T06:21:25.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:21:25.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlXQZmZwaNDBAAzS4iquvaoEIprDmF9cH70g== 2024-03-29T06:21:25.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlXQZmZwaNDBAAzS4iquvaoEIprDmF9cH70g=="}' 2024-03-29T06:21:25.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6074cc0-f2e6-4844-a9c4-b03fb873edf9 -i td/osd-scrub-repair/4/new.json 2024-03-29T06:21:25.554 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:21:25.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-29T06:21:25.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlXQZmZwaNDBAAzS4iquvaoEIprDmF9cH70g== --osd-uuid a6074cc0-f2e6-4844-a9c4-b03fb873edf9 2024-03-29T06:21:25.609 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:25.605+0000 7ffa6db907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:21:25.610 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:25.609+0000 7ffa6db907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:21:25.615 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:25.609+0000 7ffa6db907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:21:25.616 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:25.609+0000 7ffa6db907c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-29T06:21:25.616 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:25.609+0000 7ffa6db907c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-29T06:21:25.616 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:25.609+0000 7ffa6db907c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-29T06:21:25.714 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:25.713+0000 7ffa6db907c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-29T06:21:32.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-29T06:21:32.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:21:32.715 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd4 key to auth repository 2024-03-29T06:21:32.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-29T06:21:32.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:21:33.110 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.4 2024-03-29T06:21:33.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-29T06:21:33.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:21:33.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:21:33.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:21:33.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:21:33.144 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:33.140+0000 7fcf8eadc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:21:33.170 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:33.168+0000 7fcf8eadc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:21:33.171 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:33.168+0000 7fcf8eadc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:21:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-29T06:21:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:21:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-29T06:21:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:21:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:21:33.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:33.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:21:33.442 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:21:33.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:33.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:33.719 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:33.716+0000 7fcf8eadc7c0 -1 Falling back to public interface 2024-03-29T06:21:33.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:34.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:34.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:34.795 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:21:34.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:21:34.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:34.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:35.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:36.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:36.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:36.105 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:21:36.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:21:36.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:36.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:36.397 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:21:36.392+0000 7fcf8eadc7c0 -1 osd.4 0 log_to_monitors true 2024-03-29T06:21:36.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:37.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:37.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:37.421 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:21:37.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:21:37.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:37.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:37.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:38.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:38.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:38.748 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:21:38.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:21:38.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:38.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:39.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:40.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:40.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:40.141 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:21:40.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:21:40.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:40.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:40.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:41.715 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:21:41.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:21:41.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:41.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:42.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:43.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:43.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:43.092 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:21:43.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:21:43.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:43.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:43.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:44.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:44.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:44.584 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:21:44.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:21:44.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:44.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:45.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:46.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:46.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:46.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:21:46.089 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:21:46.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:46.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:46.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:47.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:47.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:47.730 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:21:47.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:21:47.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:47.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:48.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:49.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:49.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:49.085 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:21:49.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:21:49.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:49.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:50.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:50.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:50.491 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:21:50.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:21:50.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:50.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:50.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:51.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:51.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:51.831 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:21:51.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:21:51.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:51.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:52.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:53.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:53.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:53.429 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:21:53.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:21:53.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:53.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:54.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:55.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:55.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:55.171 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:21:55.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:21:55.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:55.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:55.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:56.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:56.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:56.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:21:56.698 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:21:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:57.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:21:58.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:21:58.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:21:58.675 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:21:58.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:21:58.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:21:58.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:21:59.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:00.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:00.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:00.129 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:22:00.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:22:00.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:00.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:22:00.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:01.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:01.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:01.626 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:22:01.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:22:01.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:01.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:22:02.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:03.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:03.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:03.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:22:03.321 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:22:03.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:03.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-29T06:22:03.642 INFO:tasks.workunit.client.0.smithi123.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2560062594,v1:127.0.0.1:6835/2560062594] [v2:127.0.0.1:6836/2560062594,v1:127.0.0.1:6837/2560062594] exists,up a6074cc0-f2e6-4844-a9c4-b03fb873edf9 2024-03-29T06:22:03.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:22:03.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:22:03.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:22:03.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:22:03.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-29T06:22:03.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:22:03.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:22:03.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:22:03.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:22:03.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:22:03.649 INFO:tasks.workunit.client.0.smithi123.stdout:add osd5 eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:22:03.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 eedfcd02-0949-4940-b613-b9262733805d' 2024-03-29T06:22:03.649 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:22:03.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALXgZm1DL9JxAAsnigg1BH0vcj8qWfaJn5UQ== 2024-03-29T06:22:03.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALXgZm1DL9JxAAsnigg1BH0vcj8qWfaJn5UQ=="}' 2024-03-29T06:22:03.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eedfcd02-0949-4940-b613-b9262733805d -i td/osd-scrub-repair/5/new.json 2024-03-29T06:22:04.063 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:22:04.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-29T06:22:04.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALXgZm1DL9JxAAsnigg1BH0vcj8qWfaJn5UQ== --osd-uuid eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:22:04.118 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:04.116+0000 7f1ed4a327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:04.131 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:04.128+0000 7f1ed4a327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:04.132 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:04.128+0000 7f1ed4a327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:04.133 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:04.132+0000 7f1ed4a327c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-29T06:22:04.133 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:04.132+0000 7f1ed4a327c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-29T06:22:04.133 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:04.132+0000 7f1ed4a327c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-29T06:22:04.239 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:04.236+0000 7f1ed4a327c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-29T06:22:11.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-29T06:22:11.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:22:11.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-29T06:22:11.314 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd5 key to auth repository 2024-03-29T06:22:11.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:22:11.710 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.5 2024-03-29T06:22:11.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-29T06:22:11.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:22:11.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:22:11.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:22:11.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:22:11.743 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:11.740+0000 7ff47e1f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:11.782 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:11.780+0000 7ff47e1f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:11.783 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:11.780+0000 7ff47e1f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:22:12.052 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:22:12.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:12.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:12.339 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:12.336+0000 7ff47e1f77c0 -1 Falling back to public interface 2024-03-29T06:22:12.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:13.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:13.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:13.398 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:22:13.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:22:13.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:13.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:13.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:14.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:14.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:14.718 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:22:14.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:22:14.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:14.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:15.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:15.105 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:15.104+0000 7ff47e1f77c0 -1 osd.5 0 log_to_monitors true 2024-03-29T06:22:16.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:16.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:16.034 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:22:16.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:22:16.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:16.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:16.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:17.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:17.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:17.454 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:22:17.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:22:17.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:17.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:17.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:18.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:18.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:18.771 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:22:18.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:22:18.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:18.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:19.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:20.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:20.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:20.087 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:22:20.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:22:20.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:20.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:20.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:21.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:21.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:21.411 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:22:21.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:22:21.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:21.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:21.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:22.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:22.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:22.718 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:22:22.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:22:22.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:22.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:23.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:24.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:24.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:24.074 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:22:24.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:22:24.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:24.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:24.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:25.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:25.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:25.394 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:22:25.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:22:25.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:25.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:25.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:26.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:26.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:26.715 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:22:26.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:22:26.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:26.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:27.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:28.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:28.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:28.087 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:22:28.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:22:28.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:28.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:28.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:29.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:29.441 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:22:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:22:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:29.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:29.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:30.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:30.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:30.765 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:22:30.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:22:30.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:30.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:31.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:32.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:32.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:32.218 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:22:32.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:22:32.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:32.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:33.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:34.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:34.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:34.256 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:22:34.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:22:34.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:34.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:35.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:36.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:36.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:36.032 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:22:36.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:22:36.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:36.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:36.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:37.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:37.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:37.356 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:22:37.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:22:37.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:37.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:38.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:39.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:39.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:39.138 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:22:39.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:22:39.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:39.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:39.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:40.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:40.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:40.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:22:40.607 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:22:40.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:40.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:41.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:42.053 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:22:42.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:22:42.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:42.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:22:42.409 INFO:tasks.workunit.client.0.smithi123.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/277916344,v1:127.0.0.1:6843/277916344] [v2:127.0.0.1:6844/277916344,v1:127.0.0.1:6845/277916344] exists,up eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:22:42.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:22:42.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:22:42.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:22:42.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:22:42.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-29T06:22:42.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:22:42.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:22:42.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:22:42.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3af1f669-560e-4631-9b7f-90d0bcbdfcb3 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 3af1f669-560e-4631-9b7f-90d0bcbdfcb3' 2024-03-29T06:22:42.419 INFO:tasks.workunit.client.0.smithi123.stdout:add osd6 3af1f669-560e-4631-9b7f-90d0bcbdfcb3 2024-03-29T06:22:42.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:22:42.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyXgZm7acEGhAAGPdRr4NOPr+Mbf/h32RvpA== 2024-03-29T06:22:42.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyXgZm7acEGhAAGPdRr4NOPr+Mbf/h32RvpA=="}' 2024-03-29T06:22:42.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3af1f669-560e-4631-9b7f-90d0bcbdfcb3 -i td/osd-scrub-repair/6/new.json 2024-03-29T06:22:42.917 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:22:42.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-29T06:22:42.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyXgZm7acEGhAAGPdRr4NOPr+Mbf/h32RvpA== --osd-uuid 3af1f669-560e-4631-9b7f-90d0bcbdfcb3 2024-03-29T06:22:42.975 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:42.971+0000 7f6605e837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:43.027 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:43.023+0000 7f6605e837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:43.028 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:43.027+0000 7f6605e837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:43.029 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:43.027+0000 7f6605e837c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-29T06:22:43.029 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:43.027+0000 7f6605e837c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-29T06:22:43.029 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:43.027+0000 7f6605e837c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-29T06:22:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:43.107+0000 7f6605e837c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-29T06:22:49.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-29T06:22:49.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:22:49.223 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd6 key to auth repository 2024-03-29T06:22:49.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-29T06:22:49.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:22:49.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-29T06:22:49.612 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.6 2024-03-29T06:22:49.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:22:49.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:22:49.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:22:49.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:22:49.643 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:49.639+0000 7f14ba85f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:49.645 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:49.643+0000 7f14ba85f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:49.646 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:49.643+0000 7f14ba85f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:22:49.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-29T06:22:49.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:22:49.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-29T06:22:49.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:22:49.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:22:49.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:49.946 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:22:49.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:22:49.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:49.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:50.226 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:50.223+0000 7f14ba85f7c0 -1 Falling back to public interface 2024-03-29T06:22:50.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:51.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:51.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:51.308 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:22:51.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:22:51.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:51.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:51.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:52.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:52.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:52.618 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:22:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:22:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:52.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:52.838 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:22:52.835+0000 7f14ba85f7c0 -1 osd.6 0 log_to_monitors true 2024-03-29T06:22:52.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:53.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:53.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:53.979 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:22:53.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:22:53.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:53.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:54.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:55.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:55.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:55.301 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:22:55.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:22:55.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:55.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:55.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:56.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:56.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:56.627 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:22:56.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:22:56.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:56.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:57.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:58.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:58.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:58.099 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:22:58.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:22:58.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:58.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:58.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:22:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:22:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:22:59.414 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:22:59.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:22:59.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:22:59.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:22:59.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:00.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:00.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:00.798 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:23:00.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:23:00.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:00.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:01.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:02.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:02.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:02.140 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:23:02.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:23:02.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:02.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:02.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:03.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:03.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:03.456 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:23:03.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:23:03.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:03.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:03.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:04.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:04.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:04.777 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:23:04.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:23:04.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:04.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:05.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:06.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:06.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:06.151 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:23:06.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:23:06.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:06.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:06.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:07.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:07.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:07.475 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:23:07.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:23:07.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:07.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:08.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:09.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:09.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:09.150 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:23:09.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:23:09.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:09.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:09.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:10.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:10.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:10.937 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:23:10.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:23:10.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:10.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:11.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:12.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:12.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:12.340 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:23:12.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:23:12.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:12.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:12.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:13.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:13.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:13.873 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:23:13.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:23:13.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:13.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:14.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:15.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:15.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:15.403 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:23:15.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:23:15.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:15.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:15.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:16.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:16.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:16.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:23:16.944 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:23:16.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:16.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:17.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:18.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:18.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:18.784 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:23:18.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:23:18.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:18.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:19.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:20.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:20.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:20.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:23:20.105 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:23:20.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:20.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-29T06:23:20.448 INFO:tasks.workunit.client.0.smithi123.stdout:osd.6 up in weight 1 up_from 34 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1103660569,v1:127.0.0.1:6851/1103660569] [v2:127.0.0.1:6852/1103660569,v1:127.0.0.1:6853/1103660569] exists,up 3af1f669-560e-4631-9b7f-90d0bcbdfcb3 2024-03-29T06:23:20.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:23:20.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:23:20.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:23:20.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-29T06:23:20.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-29T06:23:20.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:23:20.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:23:20.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:23:20.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:23:20.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:23:20.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:23:20.453 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:23:20.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3efe590-2608-4895-b58f-0d59d949309a 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 f3efe590-2608-4895-b58f-0d59d949309a' 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stdout:add osd7 f3efe590-2608-4895-b58f-0d59d949309a 2024-03-29T06:23:20.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:23:20.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYXgZm4dRpHBAAHRvF1uIi8WUOHct4vFvSOQ== 2024-03-29T06:23:20.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYXgZm4dRpHBAAHRvF1uIi8WUOHct4vFvSOQ=="}' 2024-03-29T06:23:20.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3efe590-2608-4895-b58f-0d59d949309a -i td/osd-scrub-repair/7/new.json 2024-03-29T06:23:20.857 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:23:20.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-29T06:23:20.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYXgZm4dRpHBAAHRvF1uIi8WUOHct4vFvSOQ== --osd-uuid f3efe590-2608-4895-b58f-0d59d949309a 2024-03-29T06:23:20.908 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:20.906+0000 7f58baa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:20.951 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:20.950+0000 7f58baa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:20.952 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:20.954+0000 7f58baa407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:20.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:20.954+0000 7f58baa407c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-29T06:23:20.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:20.954+0000 7f58baa407c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-29T06:23:20.953 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:20.954+0000 7f58baa407c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-29T06:23:21.051 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:21.050+0000 7f58baa407c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-29T06:23:28.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-29T06:23:28.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:23:28.251 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd7 key to auth repository 2024-03-29T06:23:28.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-29T06:23:28.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:23:28.643 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.7 2024-03-29T06:23:28.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-29T06:23:28.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:23:28.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:23:28.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:23:28.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:23:28.675 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:28.674+0000 7fd18330c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:28.703 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:28.702+0000 7fd18330c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:28.705 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:28.706+0000 7fd18330c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:28.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-29T06:23:28.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:23:28.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-29T06:23:28.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:23:28.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:23:28.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:28.987 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:23:28.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:23:28.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:28.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:29.255 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:29.254+0000 7fd18330c7c0 -1 Falling back to public interface 2024-03-29T06:23:29.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:30.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:30.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:30.342 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:23:30.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:23:30.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:30.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:30.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:31.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:31.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:31.712 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:23:31.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:23:31.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:31.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:31.967 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:31.966+0000 7fd18330c7c0 -1 osd.7 0 log_to_monitors true 2024-03-29T06:23:32.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:33.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:33.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:33.035 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:23:33.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:23:33.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:33.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:33.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:34.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:34.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:34.352 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:23:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:23:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:34.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:35.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:35.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:35.673 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:23:35.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:23:35.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:35.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:36.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:37.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:37.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:37.151 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:23:37.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:23:37.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:37.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:37.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:38.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:38.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:38.471 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:23:38.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:23:38.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:38.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:38.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:39.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:39.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:39.901 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:23:39.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:23:39.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:39.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:40.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:41.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:41.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:41.212 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:23:41.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:23:41.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:41.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:41.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:42.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:42.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:42.523 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:23:42.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:23:42.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:42.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:42.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:43.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:43.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:43.921 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:23:43.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:23:43.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:43.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:44.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:45.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:45.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:45.238 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:23:45.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:23:45.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:45.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:45.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:46.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:46.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:46.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:23:46.547 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:23:46.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:46.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:47.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:48.380 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:23:48.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:48.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:48.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:23:48.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:48.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:49.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:50.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:50.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:50.517 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:23:50.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:23:50.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:50.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:51.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:52.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:52.090 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:52.091 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:23:52.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:23:52.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:52.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:52.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:53.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:53.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:53.597 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:23:53.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:23:53.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:53.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:54.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:55.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:55.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:55.276 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:23:55.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:23:55.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:55.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:55.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:56.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:56.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:56.627 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:23:56.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:23:56.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:56.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:57.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:23:58.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:23:58.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:23:58.342 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:23:58.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:23:58.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:23:58.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-29T06:23:58.714 INFO:tasks.workunit.client.0.smithi123.stdout:osd.7 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/274114350,v1:127.0.0.1:6859/274114350] [v2:127.0.0.1:6860/274114350,v1:127.0.0.1:6861/274114350] exists,up f3efe590-2608-4895-b58f-0d59d949309a 2024-03-29T06:23:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:23:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:23:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:23:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:23:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-29T06:23:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:23:58.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:23:58.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-29T06:23:58.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:23:58.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-29T06:23:58.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:23:58.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:23:58.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:23:58.723 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:23:58.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-29T06:23:58.725 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:23:58.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=843491e1-bf1d-4c18-915f-9a38e8a8755f 2024-03-29T06:23:58.725 INFO:tasks.workunit.client.0.smithi123.stdout:add osd8 843491e1-bf1d-4c18-915f-9a38e8a8755f 2024-03-29T06:23:58.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 843491e1-bf1d-4c18-915f-9a38e8a8755f' 2024-03-29T06:23:58.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:23:58.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+XgZmqqkyLBAAxHIq2nxdM8+ffamtk72wVA== 2024-03-29T06:23:58.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+XgZmqqkyLBAAxHIq2nxdM8+ffamtk72wVA=="}' 2024-03-29T06:23:58.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 843491e1-bf1d-4c18-915f-9a38e8a8755f -i td/osd-scrub-repair/8/new.json 2024-03-29T06:23:59.132 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:23:59.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-29T06:23:59.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+XgZmqqkyLBAAxHIq2nxdM8+ffamtk72wVA== --osd-uuid 843491e1-bf1d-4c18-915f-9a38e8a8755f 2024-03-29T06:23:59.184 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:59.185+0000 7f86f53ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:59.242 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:59.241+0000 7f86f53ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:59.244 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:59.245+0000 7f86f53ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:23:59.245 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:59.245+0000 7f86f53ee7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-29T06:23:59.245 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:59.245+0000 7f86f53ee7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-29T06:23:59.245 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:59.245+0000 7f86f53ee7c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-29T06:23:59.320 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:23:59.317+0000 7f86f53ee7c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-29T06:24:06.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-29T06:24:06.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:24:06.806 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd8 key to auth repository 2024-03-29T06:24:06.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-29T06:24:06.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:24:07.203 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.8 2024-03-29T06:24:07.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-29T06:24:07.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:24:07.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:24:07.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:24:07.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:24:07.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:07.229+0000 7f67a97657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:07.232 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:07.233+0000 7f67a97657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:07.234 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:07.233+0000 7f67a97657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:07.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-29T06:24:07.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:24:07.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-29T06:24:07.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:24:07.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:24:07.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:07.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:24:07.565 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:24:07.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:07.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:07.779 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:07.777+0000 7f67a97657c0 -1 Falling back to public interface 2024-03-29T06:24:07.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:08.903 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:24:08.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:24:08.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:08.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:09.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:10.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:10.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:10.222 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:24:10.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:24:10.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:10.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:10.527 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:10.525+0000 7f67a97657c0 -1 osd.8 0 log_to_monitors true 2024-03-29T06:24:10.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:11.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:11.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:11.536 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:24:11.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:24:11.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:11.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:12.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:13.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:13.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:13.045 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:24:13.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:24:13.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:13.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:13.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:14.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:14.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:14.456 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:24:14.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:24:14.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:14.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:14.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:15.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:15.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:15.773 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:24:15.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:24:15.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:15.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:16.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:17.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:17.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:17.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:24:17.199 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:24:17.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:17.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:17.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:18.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:18.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:18.647 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:24:18.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:24:18.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:18.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:18.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:19.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:19.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:19.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:24:19.999 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:24:19.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:19.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:20.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:21.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:21.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:21.335 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:24:21.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:24:21.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:21.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:21.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:22.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:22.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:22.640 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:24:22.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:24:22.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:22.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:22.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:23.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:23.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:23.960 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:24:23.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:24:23.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:23.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:24.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:25.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:25.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:25.363 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:24:25.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:24:25.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:25.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:25.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:26.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:26.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:26.911 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:24:26.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:24:26.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:26.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:27.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:28.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:28.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:28.515 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:24:28.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:24:28.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:28.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:29.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:30.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:30.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:30.031 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:24:30.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:24:30.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:30.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:30.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:31.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:31.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:31.539 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:24:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:24:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:31.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:32.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:33.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:33.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:33.376 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:24:33.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:24:33.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:33.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:33.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:34.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:34.925 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:24:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:24:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:34.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:35.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:36.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:36.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:36.565 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:24:36.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:24:36.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:36.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:36.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:37.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:37.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:37.983 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:24:37.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:24:37.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:37.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-29T06:24:38.308 INFO:tasks.workunit.client.0.smithi123.stdout:osd.8 up in weight 1 up_from 44 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/4279025293,v1:127.0.0.1:6867/4279025293] [v2:127.0.0.1:6868/4279025293,v1:127.0.0.1:6869/4279025293] exists,up 843491e1-bf1d-4c18-915f-9a38e8a8755f 2024-03-29T06:24:38.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:24:38.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:24:38.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:24:38.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-29T06:24:38.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-29T06:24:38.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:24:38.313 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:24:38.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:24:38.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:24:38.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 d1adaf92-5e80-43b3-90ea-09fedafe5bbb' 2024-03-29T06:24:38.316 INFO:tasks.workunit.client.0.smithi123.stdout:add osd9 d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:24:38.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:24:38.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmXgZmd2QcFBAA5BRfVZgUsDVGiJBojuKYbA== 2024-03-29T06:24:38.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmXgZmd2QcFBAA5BRfVZgUsDVGiJBojuKYbA=="}' 2024-03-29T06:24:38.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1adaf92-5e80-43b3-90ea-09fedafe5bbb -i td/osd-scrub-repair/9/new.json 2024-03-29T06:24:38.845 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:24:38.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-29T06:24:38.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmXgZmd2QcFBAA5BRfVZgUsDVGiJBojuKYbA== --osd-uuid d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:24:38.905 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:38.904+0000 7f378bc9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:38.939 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:38.936+0000 7f378bc9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:38.941 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:38.940+0000 7f378bc9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:38.941 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:38.940+0000 7f378bc9b7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-29T06:24:38.941 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:38.940+0000 7f378bc9b7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-29T06:24:38.941 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:38.940+0000 7f378bc9b7c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-29T06:24:39.034 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:39.032+0000 7f378bc9b7c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-29T06:24:46.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-29T06:24:46.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:24:46.226 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd9 key to auth repository 2024-03-29T06:24:46.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-29T06:24:46.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:24:46.609 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.9 2024-03-29T06:24:46.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-29T06:24:46.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:24:46.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:24:46.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:24:46.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:24:46.641 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:46.640+0000 7fb5adfdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:46.643 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:46.640+0000 7fb5adfdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:46.644 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:46.644+0000 7fb5adfdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:24:46.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-29T06:24:46.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:24:46.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:24:46.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:24:46.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:24:46.922 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:24:46.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:46.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:24:46.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:46.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:47.199 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:47.196+0000 7fb5adfdc7c0 -1 Falling back to public interface 2024-03-29T06:24:47.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:48.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:48.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:48.269 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:24:48.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:24:48.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:48.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:48.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:49.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:49.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:49.580 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:24:49.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:24:49.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:49.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:49.657 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:24:49.656+0000 7fb5adfdc7c0 -1 osd.9 0 log_to_monitors true 2024-03-29T06:24:49.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:50.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:50.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:50.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:24:50.910 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:24:50.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:50.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:51.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:52.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:52.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:52.392 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:24:52.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:24:52.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:52.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:52.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:53.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:53.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:53.793 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:24:53.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:24:53.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:53.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:54.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:55.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:55.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:55.141 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:24:55.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:24:55.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:55.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:55.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:56.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:56.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:56.589 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:24:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:24:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:56.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:56.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:57.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:57.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:57.998 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:24:57.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:24:57.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:57.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:58.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:24:59.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:24:59.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:24:59.338 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:24:59.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:24:59.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:24:59.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:24:59.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:00.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:00.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:00.660 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:25:00.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:25:00.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:00.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:00.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:01.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:01.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:01.975 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:25:01.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:25:01.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:01.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:02.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:03.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:03.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:03.314 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:25:03.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:25:03.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:03.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:03.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:04.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:04.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:04.636 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:25:04.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:25:04.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:04.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:05.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:06.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:06.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:06.489 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:25:06.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:25:06.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:06.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:06.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:07.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:07.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:07.793 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:25:07.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:25:07.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:07.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:08.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:09.478 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:25:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:25:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:10.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:11.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:11.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:11.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:25:11.109 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:25:11.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:11.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:11.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:12.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:12.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:12.841 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:25:12.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:25:12.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:12.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:13.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:14.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:14.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:14.378 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:25:14.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:25:14.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:14.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:14.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:15.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:15.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:15.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:25:15.731 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:25:15.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:15.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:16.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:17.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:17.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:17.269 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:25:17.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:25:17.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:17.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:25:17.678 INFO:tasks.workunit.client.0.smithi123.stdout:osd.9 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2451272446,v1:127.0.0.1:6875/2451272446] [v2:127.0.0.1:6876/2451272446,v1:127.0.0.1:6877/2451272446] exists,up d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:25:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:25:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:25:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:25:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-29T06:25:17.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:25:18.036 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:25:18.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:25:18.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:25:18.513 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:25:18.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:25:19.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:25:23.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-29T06:25:23.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:25:23.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:25:23.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:25:23.213 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:25:23.214 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:25:23.214 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:25:23.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:25:23.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:25:23.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:25:23.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:25:23.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:25:23.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:25:23.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:25:23.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:25:23.344 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:25:23.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:25:23.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:23.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:25:23.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-03-29T06:25:23.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-03-29T06:25:23.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-03-29T06:25:23.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:23.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:25:23.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673029 2024-03-29T06:25:23.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673029 2024-03-29T06:25:23.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029' 2024-03-29T06:25:23.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:23.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:25:24.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-03-29T06:25:24.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-03-29T06:25:24.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498' 2024-03-29T06:25:24.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:24.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:25:24.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378678 2024-03-29T06:25:24.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378678 2024-03-29T06:25:24.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-81604378678' 2024-03-29T06:25:24.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:24.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:25:24.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215149 2024-03-29T06:25:24.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215149 2024-03-29T06:25:24.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-81604378678 4-103079215149' 2024-03-29T06:25:24.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:24.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:25:24.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051624 2024-03-29T06:25:24.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051624 2024-03-29T06:25:24.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-81604378678 4-103079215149 5-124554051624' 2024-03-29T06:25:24.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:24.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:25:24.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888093 2024-03-29T06:25:24.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888093 2024-03-29T06:25:24.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-81604378678 4-103079215149 5-124554051624 6-146028888093' 2024-03-29T06:25:24.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:24.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:25:24.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724568 2024-03-29T06:25:24.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724568 2024-03-29T06:25:24.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-81604378678 4-103079215149 5-124554051624 6-146028888093 7-167503724568' 2024-03-29T06:25:24.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:24.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:25:24.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561040 2024-03-29T06:25:24.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561040 2024-03-29T06:25:24.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-81604378678 4-103079215149 5-124554051624 6-146028888093 7-167503724568 8-188978561040' 2024-03-29T06:25:24.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:24.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:25:25.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397511 2024-03-29T06:25:25.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397511 2024-03-29T06:25:25.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-81604378678 4-103079215149 5-124554051624 6-146028888093 7-167503724568 8-188978561040 9-210453397511' 2024-03-29T06:25:25.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:25.125 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-03-29T06:25:25.125 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:25.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:25:25.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-03-29T06:25:25.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:25.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-03-29T06:25:25.127 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836556 2024-03-29T06:25:25.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-03-29T06:25:25.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:25:25.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-03-29T06:25:25.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:25.414 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:25.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673029 2024-03-29T06:25:25.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:25:25.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673029 2024-03-29T06:25:25.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:25.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673029 2024-03-29T06:25:25.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-03-29T06:25:25.418 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949673029 2024-03-29T06:25:25.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:25:25.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673029 -lt 42949673029 2024-03-29T06:25:25.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:25.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-03-29T06:25:25.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:25.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:25:25.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-03-29T06:25:25.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:25.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-03-29T06:25:25.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-03-29T06:25:25.707 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509498 2024-03-29T06:25:25.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:25:25.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509497 -lt 64424509498 2024-03-29T06:25:25.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:25:26.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:25:26.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:25:27.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-03-29T06:25:27.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:27.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378678 2024-03-29T06:25:27.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:27.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:25:27.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378678 2024-03-29T06:25:27.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:27.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378678 2024-03-29T06:25:27.311 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 81604378678 2024-03-29T06:25:27.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378678' 2024-03-29T06:25:27.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:25:27.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378679 -lt 81604378678 2024-03-29T06:25:27.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:27.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:27.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215149 2024-03-29T06:25:27.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:25:27.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215149 2024-03-29T06:25:27.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:27.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215149 2024-03-29T06:25:27.659 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.4 seq 103079215149 2024-03-29T06:25:27.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215149' 2024-03-29T06:25:27.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:25:27.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215149 -lt 103079215149 2024-03-29T06:25:27.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:27.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051624 2024-03-29T06:25:27.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:27.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:25:27.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051624 2024-03-29T06:25:27.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:28.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051624 2024-03-29T06:25:28.000 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.5 seq 124554051624 2024-03-29T06:25:28.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051624' 2024-03-29T06:25:28.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:25:28.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051625 -lt 124554051624 2024-03-29T06:25:28.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:28.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888093 2024-03-29T06:25:28.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:28.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:25:28.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888093 2024-03-29T06:25:28.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:28.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888093 2024-03-29T06:25:28.373 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.6 seq 146028888093 2024-03-29T06:25:28.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888093' 2024-03-29T06:25:28.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:25:28.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888093 -lt 146028888093 2024-03-29T06:25:28.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:28.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724568 2024-03-29T06:25:28.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:28.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:25:28.708 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724568 2024-03-29T06:25:28.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:28.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724568 2024-03-29T06:25:28.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724568' 2024-03-29T06:25:28.711 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.7 seq 167503724568 2024-03-29T06:25:28.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:25:29.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724568 -lt 167503724568 2024-03-29T06:25:29.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:29.032 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561040 2024-03-29T06:25:29.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:29.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:25:29.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:29.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561040 2024-03-29T06:25:29.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561040 2024-03-29T06:25:29.037 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.8 seq 188978561040 2024-03-29T06:25:29.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561040' 2024-03-29T06:25:29.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:25:29.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561040 -lt 188978561040 2024-03-29T06:25:29.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:29.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397511 2024-03-29T06:25:29.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:29.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:25:29.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397511 2024-03-29T06:25:29.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:29.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397511 2024-03-29T06:25:29.386 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.9 seq 210453397511 2024-03-29T06:25:29.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397511' 2024-03-29T06:25:29.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:25:29.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397512 -lt 210453397511 2024-03-29T06:25:29.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:25:29.714 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:25:29.714 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:25:30.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:25:30.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:25:30.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:25:30.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:25:30.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:25:30.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:25:30.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:25:30.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:25:30.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:25:30.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:25:30.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:25:30.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:25:30.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:25:30.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:25:30.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:25:30.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-03-29T06:25:30.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T06:25:30.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T06:25:30.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-29T06:25:30.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T06:25:30.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-29T06:25:31.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T06:25:31.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T06:25:32.409 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T06:25:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:25:33.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-29T06:25:33.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-29T06:25:34.022 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 2 allow_ec_overwrites to true 2024-03-29T06:25:34.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T06:25:34.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:25:34.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:25:34.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:25:34.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:25:34.046 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:25:34.047 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:25:34.047 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:25:34.047 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:25:34.047 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:25:34.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:25:34.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:25:34.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:25:34.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:25:34.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:25:34.166 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:25:34.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:25:34.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:34.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:25:34.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836559 2024-03-29T06:25:34.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836559 2024-03-29T06:25:34.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559' 2024-03-29T06:25:34.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:34.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:25:35.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-03-29T06:25:35.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-03-29T06:25:35.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033' 2024-03-29T06:25:35.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:35.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:25:35.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-03-29T06:25:35.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-03-29T06:25:35.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501' 2024-03-29T06:25:35.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:35.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:25:35.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378682 2024-03-29T06:25:35.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378682 2024-03-29T06:25:35.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501 3-81604378682' 2024-03-29T06:25:35.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:35.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:25:35.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215152 2024-03-29T06:25:35.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215152 2024-03-29T06:25:35.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501 3-81604378682 4-103079215152' 2024-03-29T06:25:35.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:35.567 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:25:35.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051627 2024-03-29T06:25:35.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051627 2024-03-29T06:25:35.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501 3-81604378682 4-103079215152 5-124554051627' 2024-03-29T06:25:35.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:35.710 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:25:35.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888096 2024-03-29T06:25:35.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888096 2024-03-29T06:25:35.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501 3-81604378682 4-103079215152 5-124554051627 6-146028888096' 2024-03-29T06:25:35.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:35.853 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:25:36.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724571 2024-03-29T06:25:36.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724571 2024-03-29T06:25:36.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501 3-81604378682 4-103079215152 5-124554051627 6-146028888096 7-167503724571' 2024-03-29T06:25:36.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:36.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:25:36.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561043 2024-03-29T06:25:36.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561043 2024-03-29T06:25:36.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501 3-81604378682 4-103079215152 5-124554051627 6-146028888096 7-167503724571 8-188978561043' 2024-03-29T06:25:36.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:36.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:25:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397514 2024-03-29T06:25:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397514 2024-03-29T06:25:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673033 2-64424509501 3-81604378682 4-103079215152 5-124554051627 6-146028888096 7-167503724571 8-188978561043 9-210453397514' 2024-03-29T06:25:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:36.297 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836559 2024-03-29T06:25:36.297 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:36.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:25:36.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836559 2024-03-29T06:25:36.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:36.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836559 2024-03-29T06:25:36.299 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836559 2024-03-29T06:25:36.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-03-29T06:25:36.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:25:36.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836559 2024-03-29T06:25:36.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:36.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-03-29T06:25:36.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:36.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:25:36.608 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:36.608 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-03-29T06:25:36.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-03-29T06:25:36.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-03-29T06:25:36.609 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949673033 2024-03-29T06:25:36.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:25:37.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673033 -lt 42949673033 2024-03-29T06:25:37.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:37.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:37.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-03-29T06:25:37.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:25:37.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-03-29T06:25:37.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:37.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-03-29T06:25:37.029 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509501 2024-03-29T06:25:37.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-03-29T06:25:37.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:25:37.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509501 -lt 64424509501 2024-03-29T06:25:37.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:37.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378682 2024-03-29T06:25:37.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:37.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:25:37.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378682 2024-03-29T06:25:37.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:37.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378682 2024-03-29T06:25:37.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378682' 2024-03-29T06:25:37.357 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.3 seq 81604378682 2024-03-29T06:25:37.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:25:37.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378682 -lt 81604378682 2024-03-29T06:25:37.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:37.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215152 2024-03-29T06:25:37.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:37.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:25:37.685 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215152 2024-03-29T06:25:37.685 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:37.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215152 2024-03-29T06:25:37.687 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.4 seq 103079215152 2024-03-29T06:25:37.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215152' 2024-03-29T06:25:37.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:25:37.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215152 -lt 103079215152 2024-03-29T06:25:37.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:37.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051627 2024-03-29T06:25:37.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:37.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:25:37.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:37.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051627 2024-03-29T06:25:37.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051627 2024-03-29T06:25:37.996 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.5 seq 124554051627 2024-03-29T06:25:37.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051627' 2024-03-29T06:25:37.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:25:38.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051628 -lt 124554051627 2024-03-29T06:25:38.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:38.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888096 2024-03-29T06:25:38.369 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:38.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:25:38.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888096 2024-03-29T06:25:38.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:38.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888096 2024-03-29T06:25:38.373 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.6 seq 146028888096 2024-03-29T06:25:38.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888096' 2024-03-29T06:25:38.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:25:38.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888096 -lt 146028888096 2024-03-29T06:25:38.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:38.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724571 2024-03-29T06:25:38.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:38.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:25:38.731 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724571 2024-03-29T06:25:38.731 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:38.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724571 2024-03-29T06:25:38.733 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.7 seq 167503724571 2024-03-29T06:25:38.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724571' 2024-03-29T06:25:38.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:25:39.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724571 -lt 167503724571 2024-03-29T06:25:39.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:39.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561043 2024-03-29T06:25:39.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:39.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:25:39.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561043 2024-03-29T06:25:39.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:39.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561043 2024-03-29T06:25:39.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561043' 2024-03-29T06:25:39.061 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.8 seq 188978561043 2024-03-29T06:25:39.062 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:25:39.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561043 -lt 188978561043 2024-03-29T06:25:39.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:39.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397514 2024-03-29T06:25:39.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:39.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:25:39.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397514 2024-03-29T06:25:39.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:39.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397514 2024-03-29T06:25:39.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397514' 2024-03-29T06:25:39.409 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.9 seq 210453397514 2024-03-29T06:25:39.410 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:25:39.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397515 -lt 210453397514 2024-03-29T06:25:39.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:25:39.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:25:39.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:25:40.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:25:40.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:25:40.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:25:40.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:25:40.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:25:40.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:25:40.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:25:40.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:25:40.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:25:40.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:25:40.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:25:40.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:25:40.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:25:40.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:25:40.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:25:40.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T06:25:40.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:25:40.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:25:41.303 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:25:41.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:25:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:25:41.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:25:41.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:25:41.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T06:25:42.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-29T06:25:42.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-29T06:25:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T06:25:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:25:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:25:42.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-29T06:25:42.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-29T06:25:42.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-29T06:25:42.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T06:25:42.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-29T06:25:42.980 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:25:42.980 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:25:43.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-29T06:25:43.305 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:25:43.305 INFO:tasks.workunit.client.0.smithi123.stderr:9 2024-03-29T06:25:43.305 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:25:43.305 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:25:43.306 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:25:43.306 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:25:43.306 INFO:tasks.workunit.client.0.smithi123.stderr:7' 2024-03-29T06:25:43.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-29T06:25:43.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-29T06:25:43.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-29T06:25:43.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:25:43.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:25:43.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:25:43.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:25:43.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:25:43.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:25:43.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:25:43.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:25:43.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:25:45.908 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:25:46.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:25:46.452 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:25:46.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:25:46.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:25:46.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:25:46.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:25:46.455 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.3 2024-03-29T06:25:46.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:25:46.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:25:46.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:25:46.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:25:46.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:25:46.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:25:46.487 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:25:46.483+0000 7ff19c2f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:25:46.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:25:46.487+0000 7ff19c2f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:25:46.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:25:46.487+0000 7ff19c2f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:25:46.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:47.050 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:25:47.047+0000 7ff19c2f17c0 -1 Falling back to public interface 2024-03-29T06:25:47.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:48.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:48.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:48.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:25:48.111 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:25:48.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:48.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:25:48.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:49.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:49.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:49.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:25:49.419 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:25:49.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:49.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:25:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:25:49.783 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:25:49.779+0000 7ff19c2f17c0 -1 osd.3 67 log_to_monitors true 2024-03-29T06:25:50.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:25:50.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:25:50.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:25:50.742 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:25:50.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:25:50.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:osd.3 up in weight 1 up_from 71 up_thru 62 down_at 68 last_clean_interval [19,67) [v2:127.0.0.1:6826/2712700969,v1:127.0.0.1:6827/2712700969] [v2:127.0.0.1:6828/2712700969,v1:127.0.0.1:6829/2712700969] exists,up 9050fb11-c4b9-4740-bfb4-e3aba24dc02c 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:25:51.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:25:51.170 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:25:51.171 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:25:51.171 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:25:51.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:25:51.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:25:51.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:25:51.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:25:51.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:25:51.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:25:51.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:25:51.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:25:51.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:25:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:25:51.711 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:25:51.711 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:25:51.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:25:51.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:51.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:25:51.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836564 2024-03-29T06:25:51.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836564 2024-03-29T06:25:51.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564' 2024-03-29T06:25:51.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:51.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:25:52.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673037 2024-03-29T06:25:52.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673037 2024-03-29T06:25:52.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037' 2024-03-29T06:25:52.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:52.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:25:52.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-03-29T06:25:52.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-03-29T06:25:52.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506' 2024-03-29T06:25:52.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:52.271 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:25:52.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-03-29T06:25:52.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-03-29T06:25:52.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506 3-304942678018' 2024-03-29T06:25:52.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:52.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:25:52.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215156 2024-03-29T06:25:52.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215156 2024-03-29T06:25:52.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506 3-304942678018 4-103079215156' 2024-03-29T06:25:52.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:52.558 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:25:52.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051632 2024-03-29T06:25:52.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051632 2024-03-29T06:25:52.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506 3-304942678018 4-103079215156 5-124554051632' 2024-03-29T06:25:52.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:52.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:25:52.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888100 2024-03-29T06:25:52.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888100 2024-03-29T06:25:52.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506 3-304942678018 4-103079215156 5-124554051632 6-146028888100' 2024-03-29T06:25:52.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:52.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:25:53.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724575 2024-03-29T06:25:53.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724575 2024-03-29T06:25:53.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506 3-304942678018 4-103079215156 5-124554051632 6-146028888100 7-167503724575' 2024-03-29T06:25:53.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:53.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:25:53.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561048 2024-03-29T06:25:53.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561048 2024-03-29T06:25:53.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506 3-304942678018 4-103079215156 5-124554051632 6-146028888100 7-167503724575 8-188978561048' 2024-03-29T06:25:53.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:25:53.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:25:53.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397519 2024-03-29T06:25:53.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397519 2024-03-29T06:25:53.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509506 3-304942678018 4-103079215156 5-124554051632 6-146028888100 7-167503724575 8-188978561048 9-210453397519' 2024-03-29T06:25:53.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:53.338 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836564 2024-03-29T06:25:53.338 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:53.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:25:53.340 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836564 2024-03-29T06:25:53.340 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:53.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836564 2024-03-29T06:25:53.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-03-29T06:25:53.342 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836564 2024-03-29T06:25:53.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:25:53.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836564 2024-03-29T06:25:53.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:53.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673037 2024-03-29T06:25:53.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:53.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:25:53.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673037 2024-03-29T06:25:53.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:53.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673037 2024-03-29T06:25:53.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-03-29T06:25:53.655 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673037 2024-03-29T06:25:53.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:25:53.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673037 2024-03-29T06:25:53.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:25:54.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:25:54.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:25:55.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673037 -lt 42949673037 2024-03-29T06:25:55.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:55.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-03-29T06:25:55.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:55.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:25:55.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-03-29T06:25:55.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:55.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-03-29T06:25:55.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-03-29T06:25:55.303 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509506 2024-03-29T06:25:55.303 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:25:55.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509506 -lt 64424509506 2024-03-29T06:25:55.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:55.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-03-29T06:25:55.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:55.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:25:55.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-03-29T06:25:55.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:55.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-03-29T06:25:55.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-03-29T06:25:55.661 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 304942678018 2024-03-29T06:25:55.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:25:55.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-03-29T06:25:55.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:55.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215156 2024-03-29T06:25:55.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:55.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:25:55.981 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215156 2024-03-29T06:25:55.981 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:55.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215156 2024-03-29T06:25:55.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215156' 2024-03-29T06:25:55.983 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 103079215156 2024-03-29T06:25:55.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:25:56.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215157 -lt 103079215156 2024-03-29T06:25:56.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:56.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051632 2024-03-29T06:25:56.342 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:56.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:25:56.344 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051632 2024-03-29T06:25:56.344 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:56.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051632 2024-03-29T06:25:56.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051632' 2024-03-29T06:25:56.346 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 124554051632 2024-03-29T06:25:56.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:25:56.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051632 -lt 124554051632 2024-03-29T06:25:56.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:56.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:56.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888100 2024-03-29T06:25:56.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:25:56.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888100 2024-03-29T06:25:56.690 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:56.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888100 2024-03-29T06:25:56.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888100' 2024-03-29T06:25:56.691 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 146028888100 2024-03-29T06:25:56.691 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:25:57.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888101 -lt 146028888100 2024-03-29T06:25:57.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:57.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724575 2024-03-29T06:25:57.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:57.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:25:57.016 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724575 2024-03-29T06:25:57.016 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:57.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724575 2024-03-29T06:25:57.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724575' 2024-03-29T06:25:57.019 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 167503724575 2024-03-29T06:25:57.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:25:57.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724576 -lt 167503724575 2024-03-29T06:25:57.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:57.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561048 2024-03-29T06:25:57.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:57.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:25:57.358 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561048 2024-03-29T06:25:57.358 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:57.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561048 2024-03-29T06:25:57.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561048' 2024-03-29T06:25:57.360 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 188978561048 2024-03-29T06:25:57.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:25:57.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561048 -lt 188978561048 2024-03-29T06:25:57.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:25:57.691 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-210453397519 2024-03-29T06:25:57.691 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:25:57.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:25:57.693 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-210453397519 2024-03-29T06:25:57.694 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:25:57.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397519 2024-03-29T06:25:57.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 210453397519' 2024-03-29T06:25:57.695 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 210453397519 2024-03-29T06:25:57.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:25:58.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397519 -lt 210453397519 2024-03-29T06:25:58.023 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:25:58.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:25:58.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:25:58.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:25:58.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:25:58.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:25:58.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:25:58.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:25:58.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:25:58.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:25:58.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:25:58.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:25:58.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:25:58.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:25:58.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:25:59.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:25:59.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:25:59.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:25:59.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:25:59.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:25:59.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:25:59.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:25:59.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:25:59.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:25:59.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:25:59.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:25:59.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:25:59.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:25:59.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:25:59.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:25:59.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:25:59.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:25:32.318908+0000 2024-03-29T06:25:59.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:26:00.176 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:26:00.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:00.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:00.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:00.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:25:32.318908+0000 '>' 2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:00.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:01.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:01.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:01.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:01.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:01.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:01.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:01.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:01.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:25:32.318908+0000 '>' 2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:01.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:02.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:02.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:02.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:02.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:02.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:02.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:02.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:03.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:25:32.318908+0000 '>' 2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:03.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:04.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:04.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:04.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:04.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:04.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:04.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:04.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:04.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:25:32.318908+0000 '>' 2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:04.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:05.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:05.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:05.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:05.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:05.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:05.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:05.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:05.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:25:32.318908+0000 '>' 2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:05.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:06.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:06.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:06.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:06.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:06.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:06.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:06.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:00.997407+0000 '>' 2024-03-29T06:25:32.318908+0000 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:26:07.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:26:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:26:07.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:26:07.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:26:07.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:26:07.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:26:07.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:26:07.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:26:07.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:26:07.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-29T06:26:09.985 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:26:09.985 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:26:09.986 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:26:10.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:26:10.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:26:10.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:26:10.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:26:10.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:26:10.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:26:10.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:26:10.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:26:10.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:26:10.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:26:10.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:26:10.533 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.9 2024-03-29T06:26:10.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:26:10.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:26:10.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:26:10.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:26:10.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:26:10.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:26:10.571 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:10.566+0000 7f1ffe70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:10.574 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:10.570+0000 7f1ffe70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:10.575 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:10.570+0000 7f1ffe70b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:10.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:26:10.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:26:10.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:26:10.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:26:10.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:26:10.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:10.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:26:10.852 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:26:10.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:10.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:11.134 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:11.130+0000 7f1ffe70b7c0 -1 Falling back to public interface 2024-03-29T06:26:11.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:12.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:12.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:12.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:26:12.203 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:26:12.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:12.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:12.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:13.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:13.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:13.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:26:13.507 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:26:13.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:13.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:13.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:13.845 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:13.842+0000 7f1ffe70b7c0 -1 osd.9 72 log_to_monitors true 2024-03-29T06:26:14.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:14.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:14.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:26:14.831 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:26:14.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:14.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:15.199 INFO:tasks.workunit.client.0.smithi123.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [49,72) [v2:127.0.0.1:6874/2134099658,v1:127.0.0.1:6875/2134099658] [v2:127.0.0.1:6876/2134099658,v1:127.0.0.1:6877/2134099658] exists,up d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:26:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:26:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:26:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:26:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:26:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:26:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:26:15.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:26:15.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:26:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:26:15.201 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:26:15.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:26:15.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:26:15.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:26:15.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:26:15.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:26:15.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:26:15.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:26:15.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:26:15.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:26:15.636 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:26:15.637 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:26:15.637 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:26:15.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:26:15.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:15.637 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:26:15.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-03-29T06:26:15.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-03-29T06:26:15.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-03-29T06:26:15.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:15.981 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:26:16.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-29T06:26:16.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-29T06:26:16.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043' 2024-03-29T06:26:16.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:16.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:26:16.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-03-29T06:26:16.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-03-29T06:26:16.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511' 2024-03-29T06:26:16.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:16.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:26:16.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678024 2024-03-29T06:26:16.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678024 2024-03-29T06:26:16.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511 3-304942678024' 2024-03-29T06:26:16.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:16.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:26:16.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215162 2024-03-29T06:26:16.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215162 2024-03-29T06:26:16.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511 3-304942678024 4-103079215162' 2024-03-29T06:26:16.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:16.901 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:26:17.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051638 2024-03-29T06:26:17.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051638 2024-03-29T06:26:17.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511 3-304942678024 4-103079215162 5-124554051638' 2024-03-29T06:26:17.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:17.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:26:17.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888106 2024-03-29T06:26:17.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888106 2024-03-29T06:26:17.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511 3-304942678024 4-103079215162 5-124554051638 6-146028888106' 2024-03-29T06:26:17.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:17.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:26:17.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724581 2024-03-29T06:26:17.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724581 2024-03-29T06:26:17.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511 3-304942678024 4-103079215162 5-124554051638 6-146028888106 7-167503724581' 2024-03-29T06:26:17.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:17.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:26:17.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561053 2024-03-29T06:26:17.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561053 2024-03-29T06:26:17.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511 3-304942678024 4-103079215162 5-124554051638 6-146028888106 7-167503724581 8-188978561053' 2024-03-29T06:26:17.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:17.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:26:17.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-29T06:26:17.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-29T06:26:17.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509511 3-304942678024 4-103079215162 5-124554051638 6-146028888106 7-167503724581 8-188978561053 9-330712481794' 2024-03-29T06:26:17.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:17.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-03-29T06:26:17.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:17.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:26:17.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-03-29T06:26:17.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:17.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-03-29T06:26:17.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-03-29T06:26:17.621 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836569 2024-03-29T06:26:17.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:26:17.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836569 2024-03-29T06:26:17.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:17.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-29T06:26:17.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:17.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:26:17.933 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-29T06:26:17.934 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:17.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-29T06:26:17.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-29T06:26:17.935 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673043 2024-03-29T06:26:17.935 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:26:18.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-03-29T06:26:18.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:18.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-03-29T06:26:18.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:18.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:26:18.292 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-03-29T06:26:18.292 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:18.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-03-29T06:26:18.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-03-29T06:26:18.294 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509511 2024-03-29T06:26:18.294 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:26:18.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509511 2024-03-29T06:26:18.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:18.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678024 2024-03-29T06:26:18.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:18.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:26:18.730 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678024 2024-03-29T06:26:18.730 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:18.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678024 2024-03-29T06:26:18.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678024' 2024-03-29T06:26:18.732 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 304942678024 2024-03-29T06:26:18.732 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:26:19.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678024 -lt 304942678024 2024-03-29T06:26:19.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:19.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215162 2024-03-29T06:26:19.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:19.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:26:19.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:19.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215162 2024-03-29T06:26:19.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215162 2024-03-29T06:26:19.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215162' 2024-03-29T06:26:19.045 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 103079215162 2024-03-29T06:26:19.045 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:26:19.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215162 -lt 103079215162 2024-03-29T06:26:19.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:19.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:19.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-124554051638 2024-03-29T06:26:19.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:26:19.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-124554051638 2024-03-29T06:26:19.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:19.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051638 2024-03-29T06:26:19.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 124554051638' 2024-03-29T06:26:19.352 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 124554051638 2024-03-29T06:26:19.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:26:19.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051638 -lt 124554051638 2024-03-29T06:26:19.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:19.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:19.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888106 2024-03-29T06:26:19.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:26:19.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888106 2024-03-29T06:26:19.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:19.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888106 2024-03-29T06:26:19.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888106' 2024-03-29T06:26:19.706 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 146028888106 2024-03-29T06:26:19.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:26:20.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888106 -lt 146028888106 2024-03-29T06:26:20.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:20.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724581 2024-03-29T06:26:20.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:20.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:26:20.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724581 2024-03-29T06:26:20.046 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:20.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724581 2024-03-29T06:26:20.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724581' 2024-03-29T06:26:20.047 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 167503724581 2024-03-29T06:26:20.047 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:26:20.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724582 -lt 167503724581 2024-03-29T06:26:20.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:20.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561053 2024-03-29T06:26:20.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:20.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:26:20.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561053 2024-03-29T06:26:20.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:20.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561053 2024-03-29T06:26:20.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561053' 2024-03-29T06:26:20.430 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 188978561053 2024-03-29T06:26:20.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:26:20.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561054 -lt 188978561053 2024-03-29T06:26:20.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:20.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-03-29T06:26:20.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:20.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:26:20.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-03-29T06:26:20.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:20.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-29T06:26:20.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-03-29T06:26:20.782 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 330712481794 2024-03-29T06:26:20.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:26:21.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-29T06:26:21.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:26:21.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:26:21.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:26:21.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:26:21.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:26:21.894 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:26:21.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:26:21.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:26:22.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:26:22.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:26:22.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:26:22.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:26:22.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:26:22.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:26:22.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:26:23.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:26:23.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:26:23.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:26:23.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:26:23.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:26:23.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:26:23.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:26:23.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-29T06:26:25.376 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:26:25.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:26:25.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:26:25.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:26:25.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:26:25.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:26:25.913 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:26:25.913 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:26:25.913 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:26:25.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:26:25.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:26:25.914 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.5 2024-03-29T06:26:25.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:26:25.915 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:26:25.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:26:25.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:26:25.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:26:25.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:26:25.948 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:25.950+0000 7fa0f21097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:25.951 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:25.950+0000 7fa0f21097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:25.952 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:25.954+0000 7fa0f21097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:26.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:26:26.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:26.490+0000 7fa0f21097c0 -1 Falling back to public interface 2024-03-29T06:26:26.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:27.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:27.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:27.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:26:27.587 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:26:27.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:27.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:26:27.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:28.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:26:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:26:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:28.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:26:29.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:29.398 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:29.398+0000 7fa0f21097c0 -1 osd.5 78 log_to_monitors true 2024-03-29T06:26:30.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:30.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:30.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:26:30.228 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:26:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:26:30.553 INFO:tasks.workunit.client.0.smithi123.stderr:osd.5 up in weight 1 up_from 82 up_thru 68 down_at 79 last_clean_interval [29,78) [v2:127.0.0.1:6842/2861353778,v1:127.0.0.1:6843/2861353778] [v2:127.0.0.1:6844/2861353778,v1:127.0.0.1:6845/2861353778] exists,up eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:26:30.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:26:30.555 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:26:30.555 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:26:30.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:26:30.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:26:30.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:26:30.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:26:30.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:26:30.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:26:30.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:26:30.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:26:30.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:26:31.328 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:26:31.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:26:31.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:31.329 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:26:31.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-03-29T06:26:31.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-03-29T06:26:31.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-03-29T06:26:31.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:31.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:26:31.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-03-29T06:26:31.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-03-29T06:26:31.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047' 2024-03-29T06:26:31.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:31.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:26:32.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509516 2024-03-29T06:26:32.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509516 2024-03-29T06:26:32.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516' 2024-03-29T06:26:32.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:32.231 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:26:32.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678028 2024-03-29T06:26:32.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678028 2024-03-29T06:26:32.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516 3-304942678028' 2024-03-29T06:26:32.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:32.377 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:26:32.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215166 2024-03-29T06:26:32.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215166 2024-03-29T06:26:32.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516 3-304942678028 4-103079215166' 2024-03-29T06:26:32.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:32.925 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:26:33.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-29T06:26:33.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-29T06:26:33.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516 3-304942678028 4-103079215166 5-352187318274' 2024-03-29T06:26:33.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:33.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:26:33.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888110 2024-03-29T06:26:33.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888110 2024-03-29T06:26:33.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516 3-304942678028 4-103079215166 5-352187318274 6-146028888110' 2024-03-29T06:26:33.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:33.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:26:33.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724585 2024-03-29T06:26:33.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724585 2024-03-29T06:26:33.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516 3-304942678028 4-103079215166 5-352187318274 6-146028888110 7-167503724585' 2024-03-29T06:26:33.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:33.379 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:26:33.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561058 2024-03-29T06:26:33.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561058 2024-03-29T06:26:33.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516 3-304942678028 4-103079215166 5-352187318274 6-146028888110 7-167503724585 8-188978561058' 2024-03-29T06:26:33.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:33.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:26:33.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-03-29T06:26:33.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-03-29T06:26:33.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673047 2-64424509516 3-304942678028 4-103079215166 5-352187318274 6-146028888110 7-167503724585 8-188978561058 9-330712481798' 2024-03-29T06:26:33.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:33.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-03-29T06:26:33.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:33.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:26:33.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-03-29T06:26:33.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:33.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-03-29T06:26:33.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-03-29T06:26:33.664 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836574 2024-03-29T06:26:33.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:26:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836574 2024-03-29T06:26:33.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:33.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-03-29T06:26:33.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:33.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:26:33.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-03-29T06:26:33.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-03-29T06:26:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-03-29T06:26:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673047 2024-03-29T06:26:33.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:26:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673047 -lt 42949673047 2024-03-29T06:26:34.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:34.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:34.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509516 2024-03-29T06:26:34.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:26:34.295 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509516 2024-03-29T06:26:34.296 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:34.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509516 2024-03-29T06:26:34.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-03-29T06:26:34.297 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509516 2024-03-29T06:26:34.297 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:26:34.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509516 2024-03-29T06:26:34.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:34.595 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678028 2024-03-29T06:26:34.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:34.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:26:34.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678028 2024-03-29T06:26:34.598 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:34.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678028 2024-03-29T06:26:34.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678028' 2024-03-29T06:26:34.600 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 304942678028 2024-03-29T06:26:34.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:26:34.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678028 -lt 304942678028 2024-03-29T06:26:34.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:34.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215166 2024-03-29T06:26:34.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:34.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:26:34.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215166 2024-03-29T06:26:34.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:34.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215166 2024-03-29T06:26:34.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215166' 2024-03-29T06:26:34.907 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 103079215166 2024-03-29T06:26:34.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:26:35.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215167 -lt 103079215166 2024-03-29T06:26:35.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:35.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318274 2024-03-29T06:26:35.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:35.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:26:35.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318274 2024-03-29T06:26:35.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:35.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-29T06:26:35.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318274' 2024-03-29T06:26:35.261 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 352187318274 2024-03-29T06:26:35.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:26:35.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-29T06:26:35.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:35.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888110 2024-03-29T06:26:35.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:35.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:26:35.599 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888110 2024-03-29T06:26:35.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:35.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888110 2024-03-29T06:26:35.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888110' 2024-03-29T06:26:35.601 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 146028888110 2024-03-29T06:26:35.602 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:26:35.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888110 -lt 146028888110 2024-03-29T06:26:35.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:35.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724585 2024-03-29T06:26:35.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:35.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:26:35.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724585 2024-03-29T06:26:35.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:35.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724585 2024-03-29T06:26:35.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724585' 2024-03-29T06:26:35.939 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 167503724585 2024-03-29T06:26:35.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:26:36.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724586 -lt 167503724585 2024-03-29T06:26:36.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:36.292 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:36.293 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561058 2024-03-29T06:26:36.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:26:36.295 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561058 2024-03-29T06:26:36.295 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561058 2024-03-29T06:26:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561058' 2024-03-29T06:26:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 188978561058 2024-03-29T06:26:36.296 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:26:36.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561058 -lt 188978561058 2024-03-29T06:26:36.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:36.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-03-29T06:26:36.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:36.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:26:36.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-03-29T06:26:36.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:36.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-03-29T06:26:36.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-03-29T06:26:36.637 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 330712481798 2024-03-29T06:26:36.637 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:26:36.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481799 -lt 330712481798 2024-03-29T06:26:36.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:26:36.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:26:36.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:26:37.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:26:37.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:26:37.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:26:37.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:26:37.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:26:37.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:26:37.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:26:37.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:26:37.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:26:37.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:26:37.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:26:37.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:26:38.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:26:38.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:26:38.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:26:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-29T06:26:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:26:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:26:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:26:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:26:38.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-29T06:26:38.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-29T06:26:38.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:26:38.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:26:38.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:38.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:38.424 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:38.424 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:26:00.997407+0000 2024-03-29T06:26:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:26:39.026 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:26:39.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:26:00.997407+0000 2024-03-29T06:26:39.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:26:39.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:26:00.997407+0000 2024-03-29T06:26:39.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:26:39.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:26:39.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:39.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:39.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:39.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:39.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:39.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:00.997407+0000 '>' 2024-03-29T06:26:00.997407+0000 2024-03-29T06:26:39.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:40.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:40.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:40.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:40.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:40.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:40.358 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:40.358 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:40.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:00.997407+0000 '>' 2024-03-29T06:26:00.997407+0000 2024-03-29T06:26:40.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:41.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:41.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:41.677 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:41.677 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:41.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:41.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:41.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:41.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:00.997407+0000 '>' 2024-03-29T06:26:00.997407+0000 2024-03-29T06:26:41.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:26:42.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:26:42.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:26:42.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:26:42.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:26:42.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:26:42.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:26:42.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:40.070145+0000 '>' 2024-03-29T06:26:00.997407+0000 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:26:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:26:43.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:26:43.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-29T06:26:45.874 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:26:45.875 INFO:tasks.workunit.client.0.smithi123.stdout:hinfo_key 2024-03-29T06:26:45.875 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:26:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:26:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:26:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:26:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:26:46.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:26:46.417 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:26:46.418 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:26:46.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:26:46.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:26:46.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:26:46.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:26:46.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:26:46.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:26:46.420 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.9 2024-03-29T06:26:46.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:26:46.421 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:26:46.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:26:46.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:26:46.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:26:46.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:26:46.453 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:46.454+0000 7f9bc22737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:46.456 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:46.458+0000 7f9bc22737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:46.458 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:46.458+0000 7f9bc22737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:26:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:26:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:26:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:26:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:26:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:26:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:46.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:26:46.756 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:26:46.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:46.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:47.002 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:47.002+0000 7f9bc22737c0 -1 Falling back to public interface 2024-03-29T06:26:47.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:48.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:48.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:48.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:26:48.063 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:26:48.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:48.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:48.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:49.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:26:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:26:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:49.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:49.593 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:26:49.594+0000 7f9bc22737c0 -1 osd.9 85 log_to_monitors true 2024-03-29T06:26:49.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:26:50.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:26:50.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:26:50.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:26:50.709 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:26:50.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:26:50.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:26:51.031 INFO:tasks.workunit.client.0.smithi123.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,85) [v2:127.0.0.1:6874/3123408238,v1:127.0.0.1:6875/3123408238] [v2:127.0.0.1:6876/3123408238,v1:127.0.0.1:6877/3123408238] exists,up d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:26:51.032 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:26:51.033 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:26:51.033 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:26:51.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:26:51.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:26:51.033 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:26:51.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:26:51.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:26:51.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:26:51.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:26:51.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:26:51.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:5 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:6 2024-03-29T06:26:51.688 INFO:tasks.workunit.client.0.smithi123.stderr:7 2024-03-29T06:26:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:8 2024-03-29T06:26:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:9' 2024-03-29T06:26:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:26:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:51.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:26:51.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-03-29T06:26:51.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-03-29T06:26:51.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-03-29T06:26:51.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:51.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:26:52.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-03-29T06:26:52.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-03-29T06:26:52.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052' 2024-03-29T06:26:52.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:52.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:26:52.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-03-29T06:26:52.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-03-29T06:26:52.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521' 2024-03-29T06:26:52.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:52.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:26:52.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678033 2024-03-29T06:26:52.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678033 2024-03-29T06:26:52.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521 3-304942678033' 2024-03-29T06:26:52.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:52.893 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:26:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215171 2024-03-29T06:26:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215171 2024-03-29T06:26:53.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521 3-304942678033 4-103079215171' 2024-03-29T06:26:53.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:53.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:26:53.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318279 2024-03-29T06:26:53.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318279 2024-03-29T06:26:53.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521 3-304942678033 4-103079215171 5-352187318279' 2024-03-29T06:26:53.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:53.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:26:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888115 2024-03-29T06:26:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888115 2024-03-29T06:26:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521 3-304942678033 4-103079215171 5-352187318279 6-146028888115' 2024-03-29T06:26:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:53.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:26:53.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724590 2024-03-29T06:26:53.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724590 2024-03-29T06:26:53.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521 3-304942678033 4-103079215171 5-352187318279 6-146028888115 7-167503724590' 2024-03-29T06:26:53.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:53.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:26:53.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561063 2024-03-29T06:26:53.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561063 2024-03-29T06:26:53.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521 3-304942678033 4-103079215171 5-352187318279 6-146028888115 7-167503724590 8-188978561063' 2024-03-29T06:26:53.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:26:53.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:26:53.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-29T06:26:53.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-29T06:26:53.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509521 3-304942678033 4-103079215171 5-352187318279 6-146028888115 7-167503724590 8-188978561063 9-377957122050' 2024-03-29T06:26:53.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:53.789 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-03-29T06:26:53.790 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:53.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:26:53.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-03-29T06:26:53.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:53.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-03-29T06:26:53.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-03-29T06:26:53.792 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836579 2024-03-29T06:26:53.793 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:26:54.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-03-29T06:26:54.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:54.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:54.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-03-29T06:26:54.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:26:54.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-03-29T06:26:54.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:54.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-03-29T06:26:54.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-03-29T06:26:54.097 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949673052 2024-03-29T06:26:54.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:26:54.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-03-29T06:26:54.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:54.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-03-29T06:26:54.424 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:54.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:26:54.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-03-29T06:26:54.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:54.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-03-29T06:26:54.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-03-29T06:26:54.427 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509521 2024-03-29T06:26:54.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:26:54.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509521 2024-03-29T06:26:54.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:54.797 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:54.797 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678033 2024-03-29T06:26:54.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:26:54.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678033 2024-03-29T06:26:54.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678033 2024-03-29T06:26:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678033' 2024-03-29T06:26:54.801 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.3 seq 304942678033 2024-03-29T06:26:54.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:26:55.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678033 -lt 304942678033 2024-03-29T06:26:55.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:55.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215171 2024-03-29T06:26:55.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:55.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:26:55.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215171 2024-03-29T06:26:55.152 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:55.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215171 2024-03-29T06:26:55.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215171' 2024-03-29T06:26:55.153 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.4 seq 103079215171 2024-03-29T06:26:55.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:26:55.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215172 -lt 103079215171 2024-03-29T06:26:55.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:55.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:55.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-352187318279 2024-03-29T06:26:55.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:26:55.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-352187318279 2024-03-29T06:26:55.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:55.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318279 2024-03-29T06:26:55.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 352187318279' 2024-03-29T06:26:55.481 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.5 seq 352187318279 2024-03-29T06:26:55.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:26:55.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318279 -lt 352187318279 2024-03-29T06:26:55.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:55.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:55.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888115 2024-03-29T06:26:55.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:26:55.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888115 2024-03-29T06:26:55.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:55.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888115 2024-03-29T06:26:55.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888115' 2024-03-29T06:26:55.821 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.6 seq 146028888115 2024-03-29T06:26:55.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:26:56.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888115 -lt 146028888115 2024-03-29T06:26:56.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:56.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724590 2024-03-29T06:26:56.106 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:56.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:26:56.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724590 2024-03-29T06:26:56.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:56.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724590 2024-03-29T06:26:56.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724590' 2024-03-29T06:26:56.110 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.7 seq 167503724590 2024-03-29T06:26:56.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:26:56.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724591 -lt 167503724590 2024-03-29T06:26:56.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:56.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561063 2024-03-29T06:26:56.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:56.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:26:56.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:56.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561063 2024-03-29T06:26:56.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561063 2024-03-29T06:26:56.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561063' 2024-03-29T06:26:56.410 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.8 seq 188978561063 2024-03-29T06:26:56.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:26:56.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561063 -lt 188978561063 2024-03-29T06:26:56.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:26:56.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-03-29T06:26:56.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:26:56.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:26:56.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-03-29T06:26:56.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:26:56.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-29T06:26:56.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-03-29T06:26:56.751 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.9 seq 377957122050 2024-03-29T06:26:56.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:26:57.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-03-29T06:26:57.072 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:26:57.072 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:26:57.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:26:57.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:26:57.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:26:57.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:26:57.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:26:57.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:26:57.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:26:57.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:26:57.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:26:57.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:26:57.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:26:57.841 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:26:57.841 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:26:58.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-29T06:26:58.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:26:58.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:26:58.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:26:58.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:26:58.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:26:58.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:26:58.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:26:58.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:26:58.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:26:58.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:26:58.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:26:58.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:26:58.311 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:26:58.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:26:58.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:26:58.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:26:58.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:26:59.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:26:59.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:26:59.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:26:59.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:26:59.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:26:59.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-29T06:26:59.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 262725"' 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 262725' 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:26:59.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 262727"' 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 262727' 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/262728: /' 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 262725 262727' 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 262725 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:26:59.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/262731: /' 2024-03-29T06:27:02.869 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:27:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.871 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:27:02.872 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:27:02.872 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:27:02.872 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:27:02.872 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:27:02.872 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:27:02.874 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:27:02.874 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:27:02.874 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:27:02.875 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: start osd.5 2024-03-29T06:27:02.876 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:27:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:27:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:27:02.879 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:27:02.879 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:27:02.879 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:27:02.879 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:27:02.879 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:27:02.881 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:27:02.881 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:27:02.881 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:27:02.881 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:27:02.881 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:27:02.881 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:27:02.882 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: start osd.9 2024-03-29T06:27:02.883 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 2024-03-29T06:27:02.925+0000 7f1ff0bfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 2024-03-29T06:27:02.929+0000 7f1ff0bfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:07.658 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 2024-03-29T06:27:02.929+0000 7f1ff0bfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 0 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 2024-03-29T06:27:03.489+0000 7f1ff0bfb7c0 -1 Falling back to public interface 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:07.659 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 1 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 2 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 2024-03-29T06:27:06.137+0000 7f1ff0bfb7c0 -1 osd.9 89 log_to_monitors true 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:07.660 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 3 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262731: osd.9 up in weight 1 up_from 94 up_thru 0 down_at 90 last_c262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 2024-03-29T06:27:02.925+0000 7f3ff15587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 2024-03-29T06:27:02.925+0000 7f3ff15587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 2024-03-29T06:27:02.929+0000 7f3ff15587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:07.661 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 0 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 2024-03-29T06:27:03.477+0000 7f3ff15587c0 -1 Falling back to public interface 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:07.662 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 1 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 2 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 2024-03-29T06:27:06.881+0000 7f3ff15587c0 -1 osd.5 89 log_to_monitors true 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:07.663 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:27:07.664 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 3 2024-03-29T06:27:07.664 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:07.664 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:08.813 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperlean_interval [88,89) [v2:127.0.0.1:6874/577461732,v1:127.0.0.1:6875/577461732] [v2:127.0.0.1:6876/577461732,v1:127.0.0.1:6877/577461732] exists,up d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:27:08.813 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:27:08.814 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 1 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 2 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 3 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 4 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 5 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 6 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 7 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 8 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: 9' 2024-03-29T06:27:08.815 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836583 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836583 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583' 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056' 2024-03-29T06:27:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.231 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/stas.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:10.231 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:10.231 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:10.231 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 4 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: osd.5 up in weight 1 up_from 95 up_thru 95 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6842/1253307830,v1:127.0.0.1:6843/1253307830] [v2:127.0.0.1:6844/1253307830,v1:127.0.0.1:6845/1253307830] exists,up eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:27:10.232 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 1 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 2 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 3 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 4 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 5 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 6 2024-03-29T06:27:10.233 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 7 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 8 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: 9' 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-03-29T06:27:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.845 INFO:tasks.workunit.client.0.smithi123.stderr:262728: ndalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:27:10.845 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-03-29T06:27:10.845 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-03-29T06:27:10.845 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525' 2024-03-29T06:27:10.845 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.845 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:27:10.845 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678037 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678037 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525 3-304942678037' 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215176 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215176 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525 3-304942678037 4-103079215176' 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-03-29T06:27:10.846 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525 3-304942678037 4-103079215176 5-408021893122' 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888120 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888120 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525 3-304942678037 4-103079215176 5-408021893122 6-146028888120' 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724595 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724595 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525 3-304942678037 4-103079215176 5-408021893122 6-146028888120 7-167503724595' 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:10.847 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:27:11.424 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.cli//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673058 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673058 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058' 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526' 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678039 2024-03-29T06:27:11.425 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678039 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526 3-304942678039' 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215177 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215177 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526 3-304942678039 4-103079215177' 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-03-29T06:27:11.426 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526 3-304942678039 4-103079215177 5-408021893123' 2024-03-29T06:27:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:27:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888121 2024-03-29T06:27:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888121 2024-03-29T06:27:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526 3-304942678039 4-103079215177 5-408021893123 6-146028888121' 2024-03-29T06:27:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:11.427 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flushent.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561067 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561067 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525 3-304942678037 4-103079215176 5-408021893122 6-146028888120 7-167503724595 8-188978561067' 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673056 2-64424509525 3-304942678037 4-103079215176 5-408021893122 6-146028888120 7-167503724595 8-188978561067 9-403726925826' 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:11.671 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836583 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836583 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836583 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.0 seq 21474836583 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836583 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:27:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.1 seq 42949673056 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673057 -lt 42949673056 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:27:12.621 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-03-29T06:27:12.621 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:12.621 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-03-29T06:27:12.621 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-03-29T06:27:12.621 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.2 seq 64424509525 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678037 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678037 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678037 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678037' 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.3 seq 304942678037 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678039 -lt 304942678037 2024-03-29T06:27:12.622 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215176 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215176 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215176 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215176' 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.4 seq 103079215176 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215177 -lt 103079215176 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-408021893122 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:27:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-408021893122 2024-03-29T06:27:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-03-29T06:27:13.392 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting o_pg_stats: seq=167503724596 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724596 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526 3-304942678039 4-103079215177 5-408021893123 6-146028888121 7-167503724596' 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561068 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561068 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526 3-304942678039 4-103079215177 5-408021893123 6-146028888121 7-167503724596 8-188978561068' 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-03-29T06:27:13.393 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673058 2-64424509526 3-304942678039 4-103079215177 5-408021893123 6-146028888121 7-167503724596 8-188978561068 9-403726925827' 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.0 seq 21474836584 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836584 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:27:13.394 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836584 2024-03-29T06:27:13.395 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:13.395 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673058 2024-03-29T06:27:13.395 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:13.395 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:27:13.395 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673058 2024-03-29T06:27:14.002 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //homesd.5 seq 408021893122' 2024-03-29T06:27:14.002 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.5 seq 408021893122 2024-03-29T06:27:14.002 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:27:14.002 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893122 2024-03-29T06:27:14.002 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888120 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888120 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888120 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888120' 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.6 seq 146028888120 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888121 -lt 146028888120 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724595 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724595 2024-03-29T06:27:14.003 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724595 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724595' 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.7 seq 167503724595 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724596 -lt 167503724595 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561067 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561067 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561067 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561067' 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.8 seq 188978561067 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:27:14.004 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561068 -lt 188978561067 2024-03-29T06:27:14.428 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_sta/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673058 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.1 seq 42949673058 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673058 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:14.429 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.2 seq 64424509526 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509526 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678039 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678039 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678039 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678039' 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.3 seq 304942678039 2024-03-29T06:27:14.430 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678039 -lt 304942678039 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215177 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215177 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215177 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215177' 2024-03-29T06:27:14.431 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.4 seq 103079215177 2024-03-29T06:27:15.541 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clonts: for s in $seqs 2024-03-29T06:27:15.541 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-403726925826 2024-03-29T06:27:15.541 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:15.541 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:27:15.541 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-403726925826 2024-03-29T06:27:15.541 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 403726925826' 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: waiting osd.9 seq 403726925826 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925828 -lt 403726925826 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:27:15.542 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:262731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:27:15.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:27:15.803 INFO:tasks.workunit.client.0.smithi123.stderr:e.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:27:15.803 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215178 -lt 103079215177 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-408021893123 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-408021893123 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.5 seq 408021893123 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893123 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888121 2024-03-29T06:27:15.804 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888121 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888121 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888121' 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.6 seq 146028888121 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888121 -lt 146028888121 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724596 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724596 2024-03-29T06:27:15.805 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:15.806 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724596 2024-03-29T06:27:15.806 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724596' 2024-03-29T06:27:15.806 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.7 seq 167503724596 2024-03-29T06:27:15.806 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:27:15.806 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724597 -lt 167503724596 2024-03-29T06:27:15.806 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:17.271 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561068 2024-03-29T06:27:17.271 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:17.271 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:27:17.271 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561068 2024-03-29T06:27:17.271 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561068 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561068' 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.8 seq 188978561068 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561069 -lt 188978561068 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-403726925827 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-403726925827 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 403726925827' 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: waiting osd.9 seq 403726925827 2024-03-29T06:27:17.272 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925828 -lt 403726925827 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:27:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:27:17.629 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:17.630 INFO:tasks.workunit.client.0.smithi123.stderr:262728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:17.630 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:27:17.630 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:27:17.630 INFO:tasks.workunit.client.0.smithi123.stderr:262728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:27:17.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:27:17.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:27:17.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 262727 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:27:17.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:27:17.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:27:17.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:27:17.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:27:17.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:27:17.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:17.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:17.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:17.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:27:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:26:40.070145+0000 2024-03-29T06:27:18.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:27:18.531 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:27:18.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:26:40.070145+0000 2024-03-29T06:27:18.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:27:18.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:26:40.070145+0000 2024-03-29T06:27:18.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:27:18.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:27:18.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:27:18.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:27:18.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:18.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:18.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:18.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:27:18.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:40.070145+0000 '>' 2024-03-29T06:26:40.070145+0000 2024-03-29T06:27:18.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:27:19.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:27:19.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:27:19.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:27:19.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:19.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:19.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:19.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:27:20.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:40.070145+0000 '>' 2024-03-29T06:26:40.070145+0000 2024-03-29T06:27:20.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:27:21.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:27:21.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:27:21.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:27:21.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:21.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:21.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:21.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:27:21.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:26:40.070145+0000 '>' 2024-03-29T06:26:40.070145+0000 2024-03-29T06:27:21.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:27:22.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:27:22.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:27:22.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:27:22.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:22.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:22.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:22.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:27:19.080608+0000 '>' 2024-03-29T06:26:40.070145+0000 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 266153"' 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 266153' 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:27:22.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 266154"' 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 266154' 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 266153 266154' 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 266153 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/266158: /' 2024-03-29T06:27:22.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/266157: /' 2024-03-29T06:27:26.317 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.317 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:27:26.317 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-29T06:27:26.317 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:27:26.318 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: _ 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: hinfo_key 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: snapset 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:27:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-29T06:27:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:27:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-29T06:27:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:27:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:27:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:266157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-29T06:27:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:266157: start osd.9 2024-03-29T06:27:26.700 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.700 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:27:26.700 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:27:26.700 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:27:26.700 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.701 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: _ 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: hinfo_key 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: snapset 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:27:26.702 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:27:26.703 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:27:26.703 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:26.704 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:27:26.705 INFO:tasks.workunit.client.0.smithi123.stderr:266158: start osd.5 2024-03-29T06:27:31.027 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 2024-03-29T06:27:26.357+0000 7f25501e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 2024-03-29T06:27:26.357+0000 7f25501e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 2024-03-29T06:27:26.361+0000 7f25501e17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-29T06:27:31.028 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 0 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 2024-03-29T06:27:26.893+0000 7f25501e17c0 -1 Falling back to public interface 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 1 2024-03-29T06:27:31.029 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 2 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 2024-03-29T06:27:29.713+0000 7f25501e17c0 -1 osd.9 97 log_to_monitors true 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.030 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:27:31.031 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 3 2024-03-29T06:27:31.031 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.031 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266157: osd.9 up in weight 1 up_from 102 up_thru 0 down_aovery_ops 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 2024-03-29T06:27:26.737+0000 7f685927a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 2024-03-29T06:27:26.793+0000 7f685927a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:31.558 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 2024-03-29T06:27:26.793+0000 7f685927a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 0 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 2024-03-29T06:27:27.333+0000 7f685927a7c0 -1 Falling back to public interface 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:31.559 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 1 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 2 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 2024-03-29T06:27:30.181+0000 7f685927a7c0 -1 osd.5 97 log_to_monitors true 2024-03-29T06:27:31.560 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:31.561 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:31.561 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:27:31.561 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 3 2024-03-29T06:27:31.561 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:31.561 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ct 98 last_clean_interval [94,97) [v2:127.0.0.1:6842/3306202610,v1:127.0.0.1:6843/3306202610] [v2:127.0.0.1:6844/3306202610,v1:127.0.0.1:6845/3306202610] exists,up d1adaf92-5e80-43b3-90ea-09fedafe5bbb 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:27:32.434 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 1 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 2 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 3 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 4 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 5 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 6 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 7 2024-03-29T06:27:32.435 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 8 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: 9' 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836590 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836590 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590' 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673063 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673063 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063' 2024-03-29T06:27:32.436 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.038 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.eph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 4 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: osd.5 up in weight 1 up_from 103 up_thru 103 down_at 98 last_clean_interval [95,97) [v2:127.0.0.1:6874/3052585997,v1:127.0.0.1:6875/3052585997] [v2:127.0.0.1:6876/3052585997,v1:127.0.0.1:6877/3052585997] exists,up eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:27:34.039 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:27:34.040 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 1 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 2 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 3 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 4 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 5 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 6 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 7 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 8 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: 9' 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836591 2024-03-29T06:27:34.041 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836591 2024-03-29T06:27:34.042 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591' 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532' 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678044 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678044 2024-03-29T06:27:34.387 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532 3-304942678044' 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215182 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215182 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532 3-304942678044 4-103079215182' 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532 3-304942678044 4-103079215182 5-442381631490' 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.388 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888126 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888126 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532 3-304942678044 4-103079215182 5-442381631490 6-146028888126' 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724602 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724602 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532 3-304942678044 4-103079215182 5-442381631490 6-146028888126 7-167503724602' 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:34.389 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:27:35.122 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cepids 2024-03-29T06:27:35.122 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:27:35.122 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673064 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673064 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064' 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533' 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678045 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678045 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533 3-304942678045' 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:27:35.123 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215184 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215184 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533 3-304942678045 4-103079215184' 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533 3-304942678045 4-103079215184 5-442381631491' 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888128 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888128 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533 3-304942678045 4-103079215184 5-442381631491 6-146028888128' 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.124 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:27:35.176 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561074 2024-03-29T06:27:35.176 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561074 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532 3-304942678044 4-103079215182 5-442381631490 6-146028888126 7-167503724602 8-188978561074' 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664194 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664194 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509532 3-304942678044 4-103079215182 5-442381631490 6-146028888126 7-167503724602 8-188978561074 9-438086664194' 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836590 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836590 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:35.177 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836590 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.0 seq 21474836590 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836591 -lt 21474836590 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673063 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673063 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673063 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-03-29T06:27:35.178 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.1 seq 42949673063 2024-03-29T06:27:35.179 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:27:35.179 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673063 2024-03-29T06:27:35.179 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:35.179 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:35.179 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-03-29T06:27:35.179 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cepht:2202: flush_pg_stats: seq=167503724603 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724603 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533 3-304942678045 4-103079215184 5-442381631491 6-146028888128 7-167503724603' 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561075 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561075 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533 3-304942678045 4-103079215184 5-442381631491 6-146028888128 7-167503724603 8-188978561075' 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:27:35.721 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664196 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664196 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509533 3-304942678045 4-103079215184 5-442381631491 6-146028888128 7-167503724603 8-188978561075 9-438086664196' 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836591 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836591 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836591 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.0 seq 21474836591 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:27:35.722 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836591 -lt 21474836591 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673064 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673064 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673064 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-03-29T06:27:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.1 seq 42949673064 2024-03-29T06:27:36.179 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221est/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.2 seq 64424509532 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678044 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:27:36.180 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678044 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678044 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678044' 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.3 seq 304942678044 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678044 -lt 304942678044 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215182 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215182 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215182 2024-03-29T06:27:36.181 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215182' 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.4 seq 103079215182 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215183 -lt 103079215182 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-442381631490 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-442381631490 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:36.182 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-03-29T06:27:37.407 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 442381631490' 2024-03-29T06:27:37.407 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.5 seq 442381631490 2024-03-29T06:27:37.407 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:27:37.407 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631490 2024-03-29T06:27:37.407 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:37.407 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888126 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888126 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888126 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888126' 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.6 seq 146028888126 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888128 -lt 146028888126 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724602 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724602 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724602 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724602' 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.7 seq 167503724602 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724603 -lt 167503724602 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561074 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561074 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561074 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561074' 2024-03-29T06:27:37.409 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.8 seq 188978561074 2024-03-29T06:27:37.410 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:27:37.410 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561075 -lt 188978561074 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673064 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673065 -lt 42949673064 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.2 seq 64424509533 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509533 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678045 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678045 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678045 2024-03-29T06:27:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678045' 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.3 seq 304942678045 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678046 -lt 304942678045 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215184 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215184 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215184 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215184' 2024-03-29T06:27:38.049 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.4 seq 103079215184 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/c0: flush_pg_stats: for s in $seqs 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664194 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664194 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664194 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664194' 2024-03-29T06:27:38.879 INFO:tasks.workunit.client.0.smithi123.stderr:266157: waiting osd.9 seq 438086664194 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664196 -lt 438086664194 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:27:38.880 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:27:38.881 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:27:38.881 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:38.881 INFO:tasks.workunit.client.0.smithi123.stderr:266157: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:38.881 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:27:38.881 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:27:38.881 INFO:tasks.workunit.client.0.smithi123.stderr:266157: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:27:38.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:27:39.482 INFO:tasks.workunit.client.0.smithi123.stderr:ephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:27:39.482 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215184 -lt 103079215184 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-442381631491 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-442381631491 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 442381631491' 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.5 seq 442381631491 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631491 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888128 2024-03-29T06:27:39.483 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888128 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888128 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888128' 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.6 seq 146028888128 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888128 -lt 146028888128 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724603 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724603 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:39.484 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724603 2024-03-29T06:27:39.485 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724603' 2024-03-29T06:27:39.485 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.7 seq 167503724603 2024-03-29T06:27:39.485 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:27:39.485 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724603 -lt 167503724603 2024-03-29T06:27:39.485 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561075 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561075 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561075 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561075' 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.8 seq 188978561075 2024-03-29T06:27:40.815 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561075 -lt 188978561075 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664196 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664196 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664196 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664196' 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: waiting osd.9 seq 438086664196 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664196 -lt 438086664196 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:27:40.816 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:27:40.817 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:27:41.166 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:41.166 INFO:tasks.workunit.client.0.smithi123.stderr:266158: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:41.166 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:27:41.166 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:27:41.166 INFO:tasks.workunit.client.0.smithi123.stderr:266158: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 266154 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:27:41.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:27:41.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:27:41.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-29T06:27:41.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-29T06:27:41.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-29T06:27:41.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269433"' 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269433' 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269435"' 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269435' 2024-03-29T06:27:41.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-29T06:27:41.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 269433 269435' 2024-03-29T06:27:41.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269436: /' 2024-03-29T06:27:41.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:27:41.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:27:41.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269433 2024-03-29T06:27:41.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:27:41.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269438: /' 2024-03-29T06:27:44.674 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.674 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:27:44.674 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:27:44.675 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:27:44.676 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:27:44.681 INFO:tasks.workunit.client.0.smithi123.stderr:269438: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:27:44.682 INFO:tasks.workunit.client.0.smithi123.stderr:269438: start osd.5 2024-03-29T06:27:44.683 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:27:44.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:27:44.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.877 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:27:44.878 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:27:44.880 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:27:44.881 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:27:44.882 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:27:44.882 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:27:44.882 INFO:tasks.workunit.client.0.smithi123.stderr:269436: start osd.3 2024-03-29T06:27:44.882 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:27:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:27:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:27:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:27:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:27:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:27:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 2024-03-29T06:27:44.713+0000 7f365408b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 2024-03-29T06:27:44.717+0000 7f365408b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 2024-03-29T06:27:44.717+0000 7f365408b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 0 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:49.278 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 2024-03-29T06:27:45.281+0000 7f365408b7c0 -1 Falling back to public interface 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 1 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.279 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 2 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 2024-03-29T06:27:48.236+0000 7f365408b7c0 -1 osd.5 105 log_to_monitors true 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 3 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.280 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpe269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 2024-03-29T06:27:44.913+0000 7f4bd5f817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 2024-03-29T06:27:44.913+0000 7f4bd5f817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 2024-03-29T06:27:44.917+0000 7f4bd5f817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:27:49.739 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 0 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 2024-03-29T06:27:45.449+0000 7f4bd5f817c0 -1 Falling back to public interface 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 1 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:27:49.740 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 2 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 2024-03-29T06:27:48.796+0000 7f4bd5f817c0 -1 osd.3 105 log_to_monitors true 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 3 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: osd.3 up in weight 1 up_from 110 up_thru 103 down_at 106 last_clean_interval [71,105) [v2:127.0.0.1:6874/1443936833,v1:127.0.0.1:6875/1443936833] [v2:127.0.0.1:6876/1443936833,v1:127.0.0.1:6877/1443936833] exists,up 9050fb11-c4b9-4740-bfb4-e3aba24dc02c 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:27:50.839 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 1 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 2 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 3 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 4 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 5 2024-03-29T06:27:50.840 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 6 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 7 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 8 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: 9' 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836595 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836595 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595' 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673069 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673069 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069' 2024-03-29T06:27:50.841 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.clienrs.sh:985: wait_for_osd: sleep 1 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 4 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:27:51.834 INFO:tasks.workunit.client.0.smithi123.stderr:269438: osd.5 up in weight 1 up_from 110 up_thru 110 down_at 106 last_clean_interval [103,105) [v2:127.0.0.1:6826/3804340531,v1:127.0.0.1:6827/3804340531] [v2:127.0.0.1:6828/3804340531,v1:127.0.0.1:6829/3804340531] exists,up eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:27:51.835 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 1 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 2 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 3 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 4 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 5 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 6 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 7 2024-03-29T06:27:51.836 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 8 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: 9' 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836597 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836597 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597' 2024-03-29T06:27:51.837 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:52.783 INFO:tasks.workunit.client.0.smithi123.stderr:26t.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:27:52.783 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-03-29T06:27:52.783 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-03-29T06:27:52.783 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537' 2024-03-29T06:27:52.783 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:52.783 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:27:52.783 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537 3-472446402562' 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215188 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215188 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537 3-472446402562 4-103079215188' 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-03-29T06:27:52.784 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537 3-472446402562 4-103079215188 5-472446402562' 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888132 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888132 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537 3-472446402562 4-103079215188 5-472446402562 6-146028888132' 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724607 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724607 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537 3-472446402562 4-103079215188 5-472446402562 6-146028888132 7-167503724607' 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:52.785 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest9438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673070 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673070 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070' 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509539 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509539 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539' 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:27:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402563 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402563 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539 3-472446402563' 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215189 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215189 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539 3-472446402563 4-103079215189' 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402563 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402563 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539 3-472446402563 4-103079215189 5-472446402563' 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:27:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888133 2024-03-29T06:27:53.067 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888133 2024-03-29T06:27:53.067 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539 3-472446402563 4-103079215189 5-472446402563 6-146028888133' 2024-03-29T06:27:53.067 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.067 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202:/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561079 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561079 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537 3-472446402562 4-103079215188 5-472446402562 6-146028888132 7-167503724607 8-188978561079' 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664200 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664200 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836595 1-42949673069 2-64424509537 3-472446402562 4-103079215188 5-472446402562 6-146028888132 7-167503724607 8-188978561079 9-438086664200' 2024-03-29T06:27:53.520 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836595 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836595 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836595 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836595' 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.0 seq 21474836595 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836597 -lt 21474836595 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673069 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:53.521 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673069 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673069 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.1 seq 42949673069 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673070 -lt 42949673069 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:53.522 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:27:53.636 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/c flush_pg_stats: seq=167503724608 2024-03-29T06:27:53.636 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724608 2024-03-29T06:27:53.636 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539 3-472446402563 4-103079215189 5-472446402563 6-146028888133 7-167503724608' 2024-03-29T06:27:53.636 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.636 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561081 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561081 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539 3-472446402563 4-103079215189 5-472446402563 6-146028888133 7-167503724608 8-188978561081' 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664201 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664201 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509539 3-472446402563 4-103079215189 5-472446402563 6-146028888133 7-167503724608 8-188978561081 9-438086664201' 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836597 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:27:53.637 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836597 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836597 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.0 seq 21474836597 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836597 -lt 21474836597 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673070 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673070 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673070 2024-03-29T06:27:53.638 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-03-29T06:27:53.639 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.1 seq 42949673070 2024-03-29T06:27:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flulone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-03-29T06:27:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-03-29T06:27:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-03-29T06:27:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.2 seq 64424509537 2024-03-29T06:27:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:27:54.589 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509538 -lt 64424509537 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-472446402562 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-472446402562 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 472446402562' 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.3 seq 472446402562 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402562 -lt 472446402562 2024-03-29T06:27:54.590 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215188 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215188 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215188 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215188' 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.4 seq 103079215188 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215190 -lt 103079215188 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-472446402562 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:27:54.591 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-472446402562 2024-03-29T06:27:54.592 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:54.592 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo sh_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673070 -lt 42949673070 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509539 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509539 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509539 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-03-29T06:27:55.010 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.2 seq 64424509539 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509539 -lt 64424509539 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-472446402563 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-472446402563 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402563 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 472446402563' 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.3 seq 472446402563 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402563 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215189 2024-03-29T06:27:55.011 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215189 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215189 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215189' 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.4 seq 103079215189 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215190 -lt 103079215189 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.012 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-472446402563 2024-03-29T06:27:55.874 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ub'waiting osd.5 seq 472446402562' 2024-03-29T06:27:55.874 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.5 seq 472446402562 2024-03-29T06:27:55.874 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:27:55.874 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402562 2024-03-29T06:27:55.874 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.874 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888132 2024-03-29T06:27:55.874 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888132 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888132 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888132' 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.6 seq 146028888132 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888133 -lt 146028888132 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724607 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724607 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.875 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724607 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724607' 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.7 seq 167503724607 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724609 -lt 167503724607 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561079 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561079 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561079 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561079' 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.8 seq 188978561079 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:27:55.876 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561081 -lt 188978561079 2024-03-29T06:27:55.958 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fluntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-472446402563 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402563 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 472446402563' 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.5 seq 472446402563 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402563 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.959 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888133 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888133 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888133 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888133' 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.6 seq 146028888133 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888133 -lt 146028888133 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724608 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724608 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724608 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724608' 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.7 seq 167503724608 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724609 -lt 167503724608 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561081 2024-03-29T06:27:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:55.962 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:27:55.962 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561081 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stush_pg_stats: for s in $seqs 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664200 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664200 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664200 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664200' 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: waiting osd.9 seq 438086664200 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664201 -lt 438086664200 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:27:57.332 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:57.333 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:27:57.334 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:27:57.334 INFO:tasks.workunit.client.0.smithi123.stderr:269436: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:27:57.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:27:57.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:27:57.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269435 2024-03-29T06:27:57.649 INFO:tasks.workunit.client.0.smithi123.stderr:ats: cut -d - -f 2 2024-03-29T06:27:57.650 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561081 2024-03-29T06:27:57.650 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561081' 2024-03-29T06:27:57.650 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.8 seq 188978561081 2024-03-29T06:27:57.650 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:27:57.650 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561081 -lt 188978561081 2024-03-29T06:27:57.651 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:27:57.651 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664201 2024-03-29T06:27:57.651 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:27:57.651 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:27:57.651 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664201 2024-03-29T06:27:57.651 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:27:57.652 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664201 2024-03-29T06:27:57.652 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664201' 2024-03-29T06:27:57.652 INFO:tasks.workunit.client.0.smithi123.stderr:269438: waiting osd.9 seq 438086664201 2024-03-29T06:27:57.652 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:27:57.652 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664202 -lt 438086664201 2024-03-29T06:27:57.653 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:27:57.653 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:57.653 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:57.653 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:27:57.653 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:27:57.654 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:27:57.654 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:27:57.654 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:27:57.654 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:27:57.654 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:27:57.655 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:27:57.655 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:27:57.655 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:27:57.655 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:27:57.655 INFO:tasks.workunit.client.0.smithi123.stderr:269438: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:27:57.655 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:27:57.656 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:27:57.656 INFO:tasks.workunit.client.0.smithi123.stderr:269438: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:27:57.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:27:57.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:27:57.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:27:57.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:27:57.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-29T06:27:57.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:27:57.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-29T06:27:57.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:27:57.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:27:57.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-29T06:27:57.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:27:57.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-29T06:27:57.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-29T06:27:57.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:27:57.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:27:57.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:57.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:57.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:57.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:27:58.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:27:19.080608+0000 2024-03-29T06:27:58.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:27:58.600 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-29T06:27:58.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:27:19.080608+0000 2024-03-29T06:27:58.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:27:58.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:27:19.080608+0000 2024-03-29T06:27:58.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:27:58.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:27:58.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:27:58.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:27:58.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:58.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:58.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:27:58.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:58.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:27:19.080608+0000 '>' 2024-03-29T06:27:19.080608+0000 2024-03-29T06:27:58.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:27:59.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:27:59.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:27:59.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:27:59.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:27:59.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:27:59.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:27:59.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:28:00.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:27:58.782640+0000 '>' 2024-03-29T06:27:19.080608+0000 2024-03-29T06:28:00.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272788"' 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272788' 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272790: /' 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272791"' 2024-03-29T06:28:00.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272791' 2024-03-29T06:28:00.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-29T06:28:00.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 272788 272791' 2024-03-29T06:28:00.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-29T06:28:00.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:28:00.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272788 2024-03-29T06:28:00.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272794: /' 2024-03-29T06:28:00.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:28:03.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:28:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: _ 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: hinfo_key 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: snapset 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-29T06:28:03.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:28:03.331 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:03.332 INFO:tasks.workunit.client.0.smithi123.stderr:272790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-29T06:28:03.333 INFO:tasks.workunit.client.0.smithi123.stderr:272790: start osd.3 2024-03-29T06:28:04.270 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:28:04.271 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: _ 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: hinfo_key 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: snapset 2024-03-29T06:28:04.272 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-29T06:28:04.273 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:28:04.273 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:28:04.273 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-29T06:28:04.273 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:28:04.273 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-29T06:28:04.273 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:28:04.273 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:28:04.275 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:28:04.275 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:28:04.275 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-29T06:28:04.275 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-29T06:28:04.275 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:28:04.275 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:28:04.276 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:28:04.277 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:28:04.277 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:28:04.277 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:28:04.277 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:28:04.277 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-29T06:28:04.277 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-29T06:28:04.277 INFO:tasks.workunit.client.0.smithi123.stderr:272794: start osd.5 2024-03-29T06:28:07.785 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 2024-03-29T06:28:03.364+0000 7f409ee0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 2024-03-29T06:28:03.364+0000 7f409ee0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 2024-03-29T06:28:03.368+0000 7f409ee0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:07.786 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 0 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 2024-03-29T06:28:03.904+0000 7f409ee0d7c0 -1 Falling back to public interface 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:07.787 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 1 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 2 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:07.788 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:28:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 2024-03-29T06:28:06.772+0000 7f409ee0d7c0 -1 osd.3 111 log_to_monitors true 2024-03-29T06:28:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:28:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 3 2024-03-29T06:28:07.789 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:09.150 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-29T06:28:09.150 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 2024-03-29T06:28:04.308+0000 7fd78add27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 2024-03-29T06:28:04.312+0000 7fd78add27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 2024-03-29T06:28:04.312+0000 7fd78add27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-29T06:28:09.151 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 0 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 2024-03-29T06:28:04.852+0000 7fd78add27c0 -1 Falling back to public interface 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:09.152 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 1 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 2 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 2024-03-29T06:28:08.148+0000 7fd78add27c0 -1 osd.5 111 log_to_monitors true 2024-03-29T06:28:09.153 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:09.154 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:09.154 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:28:09.154 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 3 2024-03-29T06:28:09.154 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:09.154 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:28:09.322 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-29T06:28:09.322 INFO:tasks.workunit.client.0.smithi123.stderr:272790: osd.3 up in weight 1 up_from 115 up_thru 115 down_at 112 last_clean_interval [110,111) [v2:127.0.0.1:6826/3699060789,v1:127.0.0.1:6827/3699060789] [v2:127.0.0.1:6828/3699060789,v1:127.0.0.1:6829/3699060789] exists,up 9050fb11-c4b9-4740-bfb4-e3aba24dc02c 2024-03-29T06:28:09.322 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:28:09.322 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:28:09.323 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:28:09.323 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:28:09.323 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:28:09.323 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:28:09.323 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:28:09.324 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:28:09.324 INFO:tasks.workunit.client.0.smithi123.stderr:272790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:28:09.324 INFO:tasks.workunit.client.0.smithi123.stderr:272790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:28:09.324 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:28:09.324 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:28:09.324 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:28:09.325 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:28:09.325 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:28:09.325 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:28:09.325 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:28:09.325 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:28:09.326 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:28:09.326 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:28:09.326 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 1 2024-03-29T06:28:09.326 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 2 2024-03-29T06:28:09.326 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 3 2024-03-29T06:28:09.326 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 4 2024-03-29T06:28:09.327 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 5 2024-03-29T06:28:09.327 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 6 2024-03-29T06:28:09.327 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 7 2024-03-29T06:28:09.327 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 8 2024-03-29T06:28:09.327 INFO:tasks.workunit.client.0.smithi123.stderr:272790: 9' 2024-03-29T06:28:09.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:28:09.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:09.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:28:09.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836601 2024-03-29T06:28:09.328 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836601 2024-03-29T06:28:09.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601' 2024-03-29T06:28:09.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:09.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:28:09.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673075 2024-03-29T06:28:09.329 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673075 2024-03-29T06:28:09.330 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075' 2024-03-29T06:28:10.993 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:10.993 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:28:10.993 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509543 2024-03-29T06:28:10.993 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509543 2024-03-29T06:28:10.993 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543' 2024-03-29T06:28:10.993 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543 3-493921239042' 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215194 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215194 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543 3-493921239042 4-103079215194' 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-03-29T06:28:10.994 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543 3-493921239042 4-103079215194 5-506806140930' 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888138 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888138 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543 3-493921239042 4-103079215194 5-506806140930 6-146028888138' 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724613 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724613 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543 3-493921239042 4-103079215194 5-506806140930 6-146028888138 7-167503724613' 2024-03-29T06:28:10.995 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:11.387 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:11.387 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:11.387 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:11.387 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:28:11.387 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 4 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: osd.5 up in weight 1 up_from 118 up_thru 118 down_at 112 last_clean_interval [110,111) [v2:127.0.0.1:6874/732250973,v1:127.0.0.1:6875/732250973] [v2:127.0.0.1:6876/732250973,v1:127.0.0.1:6877/732250973] exists,up eedfcd02-0949-4940-b613-b9262733805d 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:28:11.388 INFO:tasks.workunit.client.0.smithi123.stderr:272794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 1 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 2 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 3 2024-03-29T06:28:11.389 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 4 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 5 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 6 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 7 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 8 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: 9' 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836603 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836603 2024-03-29T06:28:11.390 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603' 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $a/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561085 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561085 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543 3-493921239042 4-103079215194 5-506806140930 6-146028888138 7-167503724613 8-188978561085' 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664206 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664206 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836601 1-42949673075 2-64424509543 3-493921239042 4-103079215194 5-506806140930 6-146028888138 7-167503724613 8-188978561085 9-438086664206' 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:12.457 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836601 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836601 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836601 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836601' 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.0 seq 21474836601 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836601 -lt 21474836601 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673075 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:28:12.458 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673075 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673075 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.1 seq 42949673075 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673075 -lt 42949673075 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509543 2024-03-29T06:28:12.459 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:12.742 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cepids 2024-03-29T06:28:12.742 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:28:12.742 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673076 2024-03-29T06:28:12.742 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673076 2024-03-29T06:28:12.742 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076' 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509544 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509544 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544' 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239043 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239043 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544 3-493921239043' 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215195 2024-03-29T06:28:12.743 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215195 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544 3-493921239043 4-103079215195' 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544 3-493921239043 4-103079215195 5-506806140931' 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888139 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888139 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544 3-493921239043 4-103079215195 5-506806140931 6-146028888139' 2024-03-29T06:28:12.744 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:12.745 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-29T06:28:13.323 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724614 2024-03-29T06:28:13.323 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724614 2024-03-29T06:28:13.323 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544 3-493921239043 4-103079215195 5-506806140931 6-146028888139 7-167503724614' 2024-03-29T06:28:13.323 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561086 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561086 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544 3-493921239043 4-103079215195 5-506806140931 6-146028888139 7-167503724614 8-188978561086' 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664207 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664207 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836603 1-42949673076 2-64424509544 3-493921239043 4-103079215195 5-506806140931 6-146028888139 7-167503724614 8-188978561086 9-438086664207' 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836603 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:13.324 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836603 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836603 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.0 seq 21474836603 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836603 -lt 21474836603 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673076 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:28:13.325 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673076 2024-03-29T06:28:13.326 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:13.326 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673076 2024-03-29T06:28:13.326 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673076' 2024-03-29T06:28:13.326 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.1 seq 42949673076 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221htest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509543 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509543 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.2 seq 64424509543 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:28:13.476 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509545 -lt 64424509543 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-493921239042 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-493921239042 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 493921239042' 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.3 seq 493921239042 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239042 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215194 2024-03-29T06:28:13.477 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215194 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215194 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215194' 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.4 seq 103079215194 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215194 -lt 103079215194 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140930 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140930 2024-03-29T06:28:13.478 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=50684: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673076 -lt 42949673076 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509544 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509544 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509544 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.2 seq 64424509544 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:28:14.725 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509545 -lt 64424509544 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-493921239043 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-493921239043 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239043 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 493921239043' 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.3 seq 493921239043 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239044 -lt 493921239043 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:14.726 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-103079215195 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-103079215195 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215195 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 103079215195' 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.4 seq 103079215195 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215196 -lt 103079215195 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:14.727 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140931 2024-03-29T06:28:14.881 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //h06140930 2024-03-29T06:28:14.881 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 506806140930' 2024-03-29T06:28:14.881 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.5 seq 506806140930 2024-03-29T06:28:14.881 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:28:14.881 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140930 -lt 506806140930 2024-03-29T06:28:14.881 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888138 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888138 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888138 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888138' 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.6 seq 146028888138 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888138 -lt 146028888138 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724613 2024-03-29T06:28:14.882 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724613 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724613 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724613' 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.7 seq 167503724613 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724615 -lt 167503724613 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561085 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561085 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:14.883 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561085 2024-03-29T06:28:14.884 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561085' 2024-03-29T06:28:14.884 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.8 seq 188978561085 2024-03-29T06:28:14.884 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1ome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140931 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 506806140931' 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.5 seq 506806140931 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140932 -lt 506806140931 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:15.717 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-146028888139 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-146028888139 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888139 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 146028888139' 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.6 seq 146028888139 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888139 -lt 146028888139 2024-03-29T06:28:15.718 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-167503724614 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-167503724614 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724614 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 167503724614' 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.7 seq 167503724614 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724615 -lt 167503724614 2024-03-29T06:28:15.719 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:15.720 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-188978561086 2024-03-29T06:28:15.720 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:15.720 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-29T06:28:15.720 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-188978561086 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush88978561087 -lt 188978561085 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664206 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664206 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664206 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664206' 2024-03-29T06:28:16.302 INFO:tasks.workunit.client.0.smithi123.stderr:272790: waiting osd.9 seq 438086664206 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664207 -lt 438086664206 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:28:16.303 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:272790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:272790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-29T06:28:16.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272791 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:_pg_stats: cut -d - -f 2 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561086 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 188978561086' 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.8 seq 188978561086 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561087 -lt 188978561086 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664207 2024-03-29T06:28:17.553 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664207 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664207 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664207' 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: waiting osd.9 seq 438086664207 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664208 -lt 438086664207 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:28:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:272794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:28:17.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-29T06:28:17.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-29T06:28:17.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-29T06:28:17.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-29T06:28:17.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-29T06:28:17.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-29T06:28:17.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:28:17.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:28:17.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:28:17.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:28:17.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:28:17.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:28:17.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:28:17.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:28:17.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:28:17.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:28:17.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:28:17.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:28:17.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:28:17.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:28:17.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:28:17.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:28:17.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:28:17.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:28:17.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:28:17.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:28:17.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:28:17.929 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:28:17.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:28:17.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:28:17.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:28:18.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:28:18.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:18.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:18.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:28:18.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:28:18.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:28:18.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:28:18.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:28:18.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:28:18.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:28:18.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:28:18.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:28:18.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:28:18.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:28:18.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:28:18.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:28:18.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:28:18.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:28:18.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:28:18.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:28:18.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:28:18.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:28:18.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:28:18.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:28:18.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:28:18.018 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:28:18.018 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:18.018 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:18.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:28:18.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:28:18.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:28:18.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:28:18.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:28:18.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:18.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:18.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:28:18.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-29T06:28:18.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:28:18.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:28:18.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:28:18.406 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:28:18.406 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:18.406 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:18.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:18.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-29T06:28:18.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:28:18.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:28:18.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:28:18.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:28:18.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:18.619 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:28:18.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:28:18.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:28:18.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:28:18.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:28:18.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:28:18.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:28:18.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:28:18.735 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:28:18.735 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:28:18.735 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:28:18.735 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:28:18.736 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:18.736 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:18.736 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:28:18.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:28:18.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:28:18.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-03-29T06:28:18.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:28:18.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:28:18.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:28:18.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:28:18.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:28:18.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:28:19.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:28:19.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:28:19.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:28:19.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:28:19.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:19.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:19.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:19.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:28:19.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:28:19.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-03-29T06:28:19.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:28:19.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:28:19.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:28:19.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:28:19.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:28:19.782 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:28:19.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:28:19.784 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:28:19.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0931d7a-91d3-4a91-9126-3e9aea86b757 2024-03-29T06:28:19.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a0931d7a-91d3-4a91-9126-3e9aea86b757' 2024-03-29T06:28:19.785 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 a0931d7a-91d3-4a91-9126-3e9aea86b757 2024-03-29T06:28:19.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:28:19.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDXwZmIebrLxAAdq4uSDh6mm5aY9UBsaCuAA== 2024-03-29T06:28:19.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDXwZmIebrLxAAdq4uSDh6mm5aY9UBsaCuAA=="}' 2024-03-29T06:28:19.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0931d7a-91d3-4a91-9126-3e9aea86b757 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:28:20.102 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:28:20.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:28:20.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDXwZmIebrLxAAdq4uSDh6mm5aY9UBsaCuAA== --osd-uuid a0931d7a-91d3-4a91-9126-3e9aea86b757 2024-03-29T06:28:20.147 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:20.144+0000 7f2ce0c167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:20.195 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:20.192+0000 7f2ce0c167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:20.197 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:20.196+0000 7f2ce0c167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:20.197 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:20.196+0000 7f2ce0c167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:28:20.198 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:20.196+0000 7f2ce0c167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:28:20.198 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:20.196+0000 7f2ce0c167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:28:20.303 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:20.300+0000 7f2ce0c167c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:28:28.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:28:28.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:28:28.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:28:28.333 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:28:28.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:28:28.715 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:28:28.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:28:28.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:28:28.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:28:28.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:28:28.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:28:28.746 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:28.744+0000 7fb8a12447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:28.749 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:28.748+0000 7fb8a12447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:28.750 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:28.748+0000 7fb8a12447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:28:29.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:28:29.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:28:29.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:28:29.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:28:29.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:28:29.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:29.020 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:28:29.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:28:29.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:29.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:29.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:29.321 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:29.320+0000 7fb8a12447c0 -1 Falling back to public interface 2024-03-29T06:28:30.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:30.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:30.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:28:30.314 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:28:30.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:30.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:30.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:31.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:31.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:31.617 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:28:31.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:28:31.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:31.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:31.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:32.255 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:28:32.252+0000 7fb8a12447c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:28:32.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:32.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:32.919 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:28:32.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:28:32.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:32.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:33.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:34.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:34.353 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:28:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:28:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:34.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:34.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:35.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:35.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:35.675 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:28:35.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:28:35.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:35.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:36.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:37.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:37.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:37.062 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:28:37.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:28:37.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:37.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:37.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:38.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:38.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:38.555 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:28:38.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:28:38.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:38.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:38.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:39.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:39.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:39.879 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:28:39.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:28:39.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:39.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:40.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:41.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:41.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:41.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:28:41.296 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:28:41.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:41.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:41.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:42.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:42.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:42.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:28:42.618 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:28:42.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:42.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:42.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:43.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:43.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:43.929 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:28:43.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:28:43.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:43.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:44.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:45.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:45.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:45.324 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:28:45.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:28:45.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:45.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:45.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:46.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:46.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:46.715 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:28:46.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:28:46.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:46.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:47.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:48.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:48.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:48.245 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:28:48.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:28:48.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:48.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:48.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:49.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:49.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:49.615 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:28:49.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:28:49.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:49.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:49.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:50.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:50.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:50.932 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:28:50.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:28:50.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:50.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:51.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:52.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:52.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:52.779 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:28:52.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:28:52.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:52.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:53.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:54.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:54.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:54.629 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:28:54.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:28:54.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:54.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:55.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:56.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:56.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:56.009 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:28:56.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:28:56.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:56.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:56.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:57.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:57.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:57.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:28:57.588 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:28:57.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:57.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:28:58.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:28:59.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:28:59.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:28:59.371 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:28:59.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:28:59.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:28:59.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:29:00.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:01.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:01.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:01.185 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:29:01.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:29:01.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:01.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:29:02.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:03.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:03.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:03.103 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:29:03.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:29:03.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:03.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:29:03.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:04.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:04.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:04.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:29:04.608 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:29:04.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:04.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:29:05.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:06.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:06.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:06.044 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:29:06.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:29:06.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:06.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:29:06.402 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2877056783,v1:127.0.0.1:6803/2877056783] [v2:127.0.0.1:6804/2877056783,v1:127.0.0.1:6805/2877056783] exists,up a0931d7a-91d3-4a91-9126-3e9aea86b757 2024-03-29T06:29:06.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:29:06.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:29:06.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:29:06.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:29:06.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:29:06.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:29:06.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:29:06.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:29:06.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:29:06.407 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:29:06.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:29:06.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:29:06.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:29:06.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:29:06.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:29:06.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc3ee566-7292-4f70-9c13-6f629542205b 2024-03-29T06:29:06.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bc3ee566-7292-4f70-9c13-6f629542205b' 2024-03-29T06:29:06.410 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 bc3ee566-7292-4f70-9c13-6f629542205b 2024-03-29T06:29:06.410 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:29:06.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyXwZm7BHRGRAAeSd7TrNFD97XsuUusEAANg== 2024-03-29T06:29:06.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyXwZm7BHRGRAAeSd7TrNFD97XsuUusEAANg=="}' 2024-03-29T06:29:06.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc3ee566-7292-4f70-9c13-6f629542205b -i td/osd-scrub-repair/1/new.json 2024-03-29T06:29:06.899 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:29:06.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:29:06.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyXwZm7BHRGRAAeSd7TrNFD97XsuUusEAANg== --osd-uuid bc3ee566-7292-4f70-9c13-6f629542205b 2024-03-29T06:29:06.956 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:06.955+0000 7f7b51f1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:29:07.076 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:07.075+0000 7f7b51f1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:29:07.078 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:07.075+0000 7f7b51f1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:29:07.079 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:07.075+0000 7f7b51f1d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:29:07.079 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:07.075+0000 7f7b51f1d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:29:07.079 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:07.075+0000 7f7b51f1d7c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:29:07.168 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:07.167+0000 7f7b51f1d7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:29:15.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:29:15.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:29:15.194 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:29:15.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:29:15.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:29:15.584 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:29:15.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:29:15.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:29:15.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:29:15.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:29:15.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:29:15.617 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:15.615+0000 7fe0de72c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:29:15.655 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:15.651+0000 7fe0de72c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:29:15.656 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:15.655+0000 7fe0de72c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:29:15.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:29:15.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:29:15.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:29:15.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:29:15.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:29:15.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:15.874 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:29:15.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:29:15.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:15.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:16.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:16.264 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:16.263+0000 7fe0de72c7c0 -1 Falling back to public interface 2024-03-29T06:29:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:17.184 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:29:17.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:29:17.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:17.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:17.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:18.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:18.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:18.495 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:29:18.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:29:18.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:18.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:18.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:19.194 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:29:19.191+0000 7fe0de72c7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:29:19.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:19.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:19.805 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:29:19.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:29:19.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:19.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:20.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:21.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:21.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:21.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:29:21.262 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:29:21.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:21.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:21.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:22.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:22.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:22.710 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:29:22.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:29:22.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:22.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:23.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:24.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:24.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:24.021 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:29:24.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:29:24.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:24.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:24.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:25.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:25.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:25.459 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:29:25.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:29:25.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:25.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:25.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:26.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:26.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:26.774 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:29:26.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:29:26.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:26.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:27.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:28.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:28.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:28.238 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:29:28.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:29:28.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:28.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:28.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:29.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:29.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:29.663 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:29:29.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:29:29.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:29.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:29.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:30.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:30.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:30.977 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:29:30.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:29:30.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:30.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:31.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:32.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:32.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:32.463 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:29:32.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:29:32.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:32.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:32.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:33.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:33.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:33.962 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:29:33.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:29:33.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:34.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:35.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:35.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:35.460 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:29:35.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:29:35.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:35.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:35.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:36.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:36.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:36.781 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:29:36.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:29:36.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:36.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:37.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:38.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:38.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:38.289 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:29:38.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:29:38.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:38.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:38.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:39.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:39.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:39.597 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:29:39.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:29:39.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:39.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:40.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:41.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:41.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:41.296 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:29:41.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:29:41.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:41.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:42.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:43.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:43.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:43.005 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:29:43.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:29:43.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:43.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:44.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:45.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:45.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:45.083 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:29:45.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:29:45.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:45.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:45.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:46.770 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:29:46.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:29:46.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:46.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:47.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:48.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:48.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:48.865 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:29:48.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:29:48.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:48.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:49.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:50.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:50.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:50.664 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:29:50.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:29:50.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:50.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:51.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:52.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:52.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:52.390 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:29:52.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:29:52.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:52.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:52.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:29:53.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:29:53.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:29:53.811 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:29:53.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:29:53.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:29:53.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:29:54.131 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2673910694,v1:127.0.0.1:6811/2673910694] [v2:127.0.0.1:6812/2673910694,v1:127.0.0.1:6813/2673910694] exists,up bc3ee566-7292-4f70-9c13-6f629542205b 2024-03-29T06:29:54.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:29:54.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:29:54.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:29:54.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-03-29T06:29:54.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:29:54.600 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:29:54.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:29:54.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:29:55.051 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:29:55.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:29:56.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:29:59.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-03-29T06:29:59.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:29:59.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:29:59.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:29:59.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:29:59.366 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:29:59.366 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:29:59.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:29:59.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:29:59.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:29:59.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:29:59.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:29:59.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:29:59.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:29:59.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:29:59.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:29:59.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:29:59.811 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:29:59.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:29:59.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:29:59.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:29:59.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-03-29T06:29:59.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-03-29T06:29:59.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-03-29T06:29:59.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:29:59.998 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:30:00.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-29T06:30:00.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-29T06:30:00.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672970' 2024-03-29T06:30:00.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:00.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-03-29T06:30:00.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:00.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:30:00.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-03-29T06:30:00.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:00.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-03-29T06:30:00.146 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836496 2024-03-29T06:30:00.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-03-29T06:30:00.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:30:00.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-03-29T06:30:00.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:30:01.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:30:01.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:30:01.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-03-29T06:30:01.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:01.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:01.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-29T06:30:01.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:30:01.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-29T06:30:01.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:01.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-29T06:30:01.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-29T06:30:01.763 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672970 2024-03-29T06:30:01.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:30:02.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-29T06:30:02.062 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:30:02.062 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:02.063 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:02.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:30:02.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:02.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:02.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:02.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:02.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:02.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:02.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:02.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:30:02.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:02.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:02.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:03.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:30:03.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:30:03.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:30:03.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-03-29T06:30:03.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:30:03.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-29T06:30:03.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-29T06:30:03.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:30:03.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:30:03.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:30:03.529 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:30:03.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:30:04.080 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:30:04.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:30:04.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:30:04.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-29T06:30:04.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-03-29T06:30:04.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-29T06:30:04.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-29T06:30:04.556 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-29T06:30:04.556 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-29T06:30:04.556 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T06:30:04.557 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-29T06:30:04.557 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:30:05.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T06:30:05.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-29T06:30:05.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T06:30:05.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-29T06:30:05.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:30:05.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-29T06:30:05.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-29T06:30:05.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T06:30:05.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:30:05.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:30:05.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:30:06.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:30:06.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-29T06:30:06.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:30:06.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:30:06.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:30:06.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:30:06.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:30:06.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-29T06:30:08.428 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-29T06:30:08.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:30:08.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:30:08.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:30:08.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:30:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:30:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:30:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:30:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:30:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:30:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:30:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:30:08.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:30:08.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:30:08.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:30:08.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:30:08.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:30:08.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:30:09.003 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:09.002+0000 7f4d9aff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:09.006 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:09.006+0000 7f4d9aff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:09.007 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:09.010+0000 7f4d9aff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:30:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:30:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:30:09.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:30:09.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:30:09.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:09.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:30:09.290 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:30:09.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:09.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:30:09.538 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:09.538+0000 7f4d9aff17c0 -1 Falling back to public interface 2024-03-29T06:30:09.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:10.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:10.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:10.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:30:10.603 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:30:10.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:10.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:30:10.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:30:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:30:11.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:11.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:30:12.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:12.543 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:12.541+0000 7f4d9aff17c0 -1 osd.0 22 log_to_monitors true 2024-03-29T06:30:13.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:13.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:13.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:30:13.232 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:30:13.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:13.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:30:13.665 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 26 up_thru 12 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/2336014098,v1:127.0.0.1:6803/2336014098] [v2:127.0.0.1:6804/2336014098,v1:127.0.0.1:6805/2336014098] exists,up a0931d7a-91d3-4a91-9126-3e9aea86b757 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:30:13.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:30:13.668 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:30:13.668 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:30:13.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:30:13.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:30:13.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:30:13.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:30:13.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:30:13.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:30:13.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:30:13.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:30:13.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:30:14.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:30:14.269 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:30:14.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:30:14.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:30:14.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:30:14.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-03-29T06:30:14.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-03-29T06:30:14.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-03-29T06:30:14.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:30:14.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:30:14.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-29T06:30:14.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-29T06:30:14.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672974' 2024-03-29T06:30:14.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:14.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-03-29T06:30:14.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:14.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:30:14.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-03-29T06:30:14.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:14.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-03-29T06:30:14.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-03-29T06:30:14.854 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 111669149698 2024-03-29T06:30:14.854 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:30:15.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-03-29T06:30:15.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:30:16.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:30:16.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:30:16.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-03-29T06:30:16.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:16.613 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-29T06:30:16.613 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:16.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:30:16.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-29T06:30:16.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:16.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-29T06:30:16.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-29T06:30:16.617 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672974 2024-03-29T06:30:16.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:30:16.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-03-29T06:30:16.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:30:16.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:16.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:17.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:30:17.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:17.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:17.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:17.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:17.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:17.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:17.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:17.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:30:17.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:17.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:17.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:17.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-29T06:30:17.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T06:30:17.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:30:17.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T06:30:17.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:30:17.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:30:18.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:30:18.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:18.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:18.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:18.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:18.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:18.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:18.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:18.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:30:18.393 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:18.393 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:18.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:18.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-29T06:30:18.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-29T06:30:18.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:30:18.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:30:18.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:30:18.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:30:18.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:30:18.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:30:18.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:30:19.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:30:19.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:30:19.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:30:19.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:30:19.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:30:19.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:30:19.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:19.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:19.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:19.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:19.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:19.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:19.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:19.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:30:19.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:19.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:19.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:20.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-29T06:30:20.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-29T06:30:20.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:30:20.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:30:20.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:30:20.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:30:20.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:30:20.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:30:20.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:30:20.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:30:20.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:30:20.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:30:20.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:30:20.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:30:20.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:30:20.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:20.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:20.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:20.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:20.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:20.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:20.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:21.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:30:21.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:21.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:21.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:21.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:30:21.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:30:21.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:30:21.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-29T06:30:21.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-29T06:30:21.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:30:21.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-29T06:30:21.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:30:21.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-29T06:30:21.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-29T06:30:21.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-29T06:30:21.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-29T06:30:21.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T06:30:21.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:30:21.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:30:21.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T06:30:22.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:29:54.974253+0000 2024-03-29T06:30:22.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-29T06:30:22.450 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-29T06:30:22.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-29T06:29:54.974253+0000 2024-03-29T06:30:22.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-29T06:30:22.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:29:54.974253+0000 2024-03-29T06:30:22.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:30:22.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:30:22.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:30:22.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T06:30:22.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T06:30:22.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:30:22.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:30:22.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T06:30:22.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:29:54.974253+0000 '>' 2024-03-29T06:29:54.974253+0000 2024-03-29T06:30:22.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:30:23.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:30:23.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:30:23.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T06:30:23.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T06:30:23.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:30:23.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:30:23.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T06:30:24.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:29:54.974253+0000 '>' 2024-03-29T06:29:54.974253+0000 2024-03-29T06:30:24.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:30:25.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:30:25.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:30:25.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T06:30:25.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T06:30:25.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:30:25.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:30:25.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:30:23.173783+0000 '>' 2024-03-29T06:29:54.974253+0000 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:30:25.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:30:25.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:30:25.413 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:30:25.413 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:30:25.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:30:25.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:30:25.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:30:25.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:30:25.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:30:25.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:30:25.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:30:25.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:30:25.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:30:25.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:30:25.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T06:30:27.733 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-29T06:30:27.733 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:30:27.734 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:30:28.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:30:28.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:30:28.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:30:28.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:30:28.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:30:28.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:30:28.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:30:28.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:30:28.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:30:28.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:30:28.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:30:28.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:30:28.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:30:28.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:30:28.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:30:28.038 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:30:28.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:30:28.038 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:30:28.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:30:28.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:30:28.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:30:28.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:30:28.072 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:28.073+0000 7fd5f595b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:28.075 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:28.077+0000 7fd5f595b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:28.076 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:28.077+0000 7fd5f595b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:28.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:30:28.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:30:28.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:30:28.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:30:28.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:30:28.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:28.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:30:28.358 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:30:28.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:28.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:28.625 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:28.625+0000 7fd5f595b7c0 -1 Falling back to public interface 2024-03-29T06:30:28.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:29.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:29.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:29.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:30:29.753 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:30:29.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:29.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:30.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:31.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:31.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:31.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:30:31.066 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:30:31.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:31.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:31.289 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:31.289+0000 7fd5f595b7c0 -1 osd.1 28 log_to_monitors true 2024-03-29T06:30:31.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:32.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:32.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:32.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:30:32.426 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:30:32.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:32.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/870970221,v1:127.0.0.1:6811/870970221] [v2:127.0.0.1:6812/870970221,v1:127.0.0.1:6813/870970221] exists,up bc3ee566-7292-4f70-9c13-6f629542205b 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:30:32.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:30:32.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:30:32.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:30:32.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:30:32.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:30:32.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:30:32.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:30:32.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:30:32.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:30:32.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:30:32.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:30:32.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:30:32.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:30:33.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:30:33.373 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:30:33.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:30:33.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:30:33.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:30:33.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149703 2024-03-29T06:30:33.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149703 2024-03-29T06:30:33.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703' 2024-03-29T06:30:33.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:30:33.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:30:33.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-03-29T06:30:33.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-03-29T06:30:33.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703 1-137438953474' 2024-03-29T06:30:33.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:33.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149703 2024-03-29T06:30:33.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:33.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:30:33.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149703 2024-03-29T06:30:33.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:33.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149703 2024-03-29T06:30:33.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149703' 2024-03-29T06:30:33.725 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 111669149703 2024-03-29T06:30:33.725 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:30:34.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149703 2024-03-29T06:30:34.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:30:35.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:30:35.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:30:35.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149703 -lt 111669149703 2024-03-29T06:30:35.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:35.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-03-29T06:30:35.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:35.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:30:35.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-03-29T06:30:35.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:35.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-03-29T06:30:35.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-03-29T06:30:35.423 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 137438953474 2024-03-29T06:30:35.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:30:35.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-03-29T06:30:35.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:30:35.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:35.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:36.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:30:36.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:36.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:36.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:36.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:36.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:36.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:36.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:36.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:30:36.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:36.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:36.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:36.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-29T06:30:36.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T06:30:36.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:30:36.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T06:30:36.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:30:36.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:30:36.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:30:36.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:36.930 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:36.930 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:36.930 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:36.930 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:36.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:36.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:37.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:30:37.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:37.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:37.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:37.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-29T06:30:37.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-29T06:30:37.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:30:37.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:30:37.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:30:37.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:30:37.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:30:37.601 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:30:37.601 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:30:38.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:30:38.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:30:38.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:30:38.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:30:38.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:30:38.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:30:38.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:38.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:38.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:38.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:38.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:38.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:38.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:38.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:30:38.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:38.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:38.532 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:38.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:30:38.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:30:38.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:30:38.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:30:39.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:30:39.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-03-29T06:30:39.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-29T06:30:39.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-29T06:30:39.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-29T06:30:39.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:30:39.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:30:39.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:30:40.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-29T06:30:42.280 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-29T06:30:42.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:30:42.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:30:42.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:30:42.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:30:42.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:30:42.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:30:42.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:30:42.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:30:42.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:30:42.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:30:42.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:30:42.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:30:42.823 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:30:42.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:30:42.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:30:42.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:30:42.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:30:42.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:30:42.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:30:42.854 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:42.853+0000 7f4ec1cd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:42.897 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:42.897+0000 7f4ec1cd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:42.899 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:42.901+0000 7f4ec1cd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:43.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:43.441 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:43.441+0000 7f4ec1cd87c0 -1 Falling back to public interface 2024-03-29T06:30:43.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:44.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:44.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:44.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:30:44.532 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:30:44.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:44.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:44.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:45.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:45.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:45.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:30:45.873 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:30:45.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:45.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:46.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:46.244 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:46.245+0000 7f4ec1cd87c0 -1 osd.1 34 log_to_monitors true 2024-03-29T06:30:47.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:47.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:47.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:30:47.196 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:30:47.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:47.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/2557722718,v1:127.0.0.1:6811/2557722718] [v2:127.0.0.1:6812/2557722718,v1:127.0.0.1:6813/2557722718] exists,up bc3ee566-7292-4f70-9c13-6f629542205b 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:30:47.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:30:47.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:30:47.516 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:30:47.516 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:30:47.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:30:47.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:30:47.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:30:47.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:30:47.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:30:47.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:30:47.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:30:47.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:30:47.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:30:48.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:30:48.062 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:30:48.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:30:48.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:30:48.062 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:30:48.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149707 2024-03-29T06:30:48.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149707 2024-03-29T06:30:48.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149707' 2024-03-29T06:30:48.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:30:48.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:30:48.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757250 2024-03-29T06:30:48.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757250 2024-03-29T06:30:48.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149707 1-163208757250' 2024-03-29T06:30:48.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:48.393 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149707 2024-03-29T06:30:48.393 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:48.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:30:48.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149707 2024-03-29T06:30:48.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:48.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149707 2024-03-29T06:30:48.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149707' 2024-03-29T06:30:48.396 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 111669149707 2024-03-29T06:30:48.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:30:48.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149707 -lt 111669149707 2024-03-29T06:30:48.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-163208757250 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-163208757250 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757250 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-03-29T06:30:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 163208757250 2024-03-29T06:30:48.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:30:48.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757250 2024-03-29T06:30:48.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:30:48.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:48.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:49.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:30:49.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:30:49.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:30:49.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:30:49.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:30:49.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:30:49.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:30:49.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:30:49.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:30:49.634 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:30:49.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:30:49.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:30:50.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:30:50.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:30:50.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:30:50.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-29T06:30:50.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-29T06:30:50.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-29T06:30:50.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-29T06:30:50.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:30:50.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-29T06:30:50.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-29T06:30:50.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-29T06:30:50.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-29T06:30:50.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T06:30:50.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:30:50.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T06:30:50.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:30:50.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:30:23.173783+0000 2024-03-29T06:30:50.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-29T06:30:50.912 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-29T06:30:50.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-29T06:30:23.173783+0000 2024-03-29T06:30:50.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-29T06:30:50.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:30:23.173783+0000 2024-03-29T06:30:50.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:30:50.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:30:50.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:30:50.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T06:30:50.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T06:30:50.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:30:50.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:30:50.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T06:30:51.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:30:23.173783+0000 '>' 2024-03-29T06:30:23.173783+0000 2024-03-29T06:30:51.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:30:52.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:30:52.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:30:52.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-29T06:30:52.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-29T06:30:52.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:30:52.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:30:52.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-29T06:30:52.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:30:51.393712+0000 '>' 2024-03-29T06:30:23.173783+0000 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:30:52.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:30:52.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:30:52.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:30:52.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:30:52.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:30:52.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:30:52.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-29T06:30:54.921 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-29T06:30:54.921 INFO:tasks.workunit.client.0.smithi123.stdout:_ 2024-03-29T06:30:54.921 INFO:tasks.workunit.client.0.smithi123.stdout:snapset 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:30:55.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:30:55.208 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:30:55.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:55.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:30:55.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:30:55.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:30:55.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:30:55.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:30:55.211 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:30:55.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:30:55.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:30:55.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:30:55.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:30:55.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:30:55.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:30:55.244 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:55.245+0000 7f3e824917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:55.247 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:55.245+0000 7f3e824917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:55.248 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:55.249+0000 7f3e824917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:30:55.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:30:55.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:30:55.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:30:55.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:30:55.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:30:55.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:55.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:30:55.521 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:30:55.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:55.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:55.781 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:55.781+0000 7f3e824917c0 -1 Falling back to public interface 2024-03-29T06:30:55.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:56.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:56.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:56.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:30:56.891 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:30:56.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:56.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:57.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:58.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:58.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:58.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:30:58.218 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:30:58.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:58.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:58.510 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:30:58.509+0000 7f3e824917c0 -1 osd.1 39 log_to_monitors true 2024-03-29T06:30:58.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:30:59.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:30:59.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:30:59.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:30:59.541 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:30:59.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:30:59.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:30:59.925 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/670074157,v1:127.0.0.1:6811/670074157] [v2:127.0.0.1:6812/670074157,v1:127.0.0.1:6813/670074157] exists,up bc3ee566-7292-4f70-9c13-6f629542205b 2024-03-29T06:30:59.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:30:59.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:30:59.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:30:59.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:30:59.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:30:59.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:30:59.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:30:59.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:30:59.927 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:30:59.927 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:30:59.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:30:59.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:30:59.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:31:00.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:31:00.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:31:00.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:31:00.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:31:00.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:31:00.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:31:00.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:31:00.623 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:31:00.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:31:00.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:31:00.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:31:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149710 2024-03-29T06:31:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149710 2024-03-29T06:31:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149710' 2024-03-29T06:31:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:31:00.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:31:01.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-29T06:31:01.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-29T06:31:01.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149710 1-184683593730' 2024-03-29T06:31:01.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:31:01.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149710 2024-03-29T06:31:01.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:31:01.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:31:01.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149710 2024-03-29T06:31:01.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:31:01.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149710 2024-03-29T06:31:01.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149710' 2024-03-29T06:31:01.093 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 111669149710 2024-03-29T06:31:01.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:31:01.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149709 -lt 111669149710 2024-03-29T06:31:01.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:31:02.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:31:02.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:31:02.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149710 -lt 111669149710 2024-03-29T06:31:02.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:31:02.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593730 2024-03-29T06:31:02.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:31:02.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:31:02.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593730 2024-03-29T06:31:02.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:31:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-29T06:31:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-03-29T06:31:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 184683593730 2024-03-29T06:31:02.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:31:03.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-29T06:31:03.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:31:03.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:31:03.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:31:03.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:31:03.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:31:03.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:31:03.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:31:03.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:31:03.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:31:03.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:31:03.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:31:03.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:31:03.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:31:03.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:31:03.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:31:04.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 4 2024-03-29T06:31:04.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:31:04.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:31:04.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:31:04.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:31:04.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:31:04.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:31:04.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:31:04.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:31:04.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:31:04.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:31:04.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:31:04.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:31:04.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:31:04.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:31:04.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:31:04.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:31:04.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:31:05.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:31:05.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:31:05.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:31:05.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-29T06:31:05.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-29T06:31:05.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:31:05.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:31:05.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:31:05.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:31:05.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:31:05.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:31:05.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:31:05.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:31:05.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:31:05.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:31:05.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:31:05.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:31:05.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:31:05.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:31:05.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:31:05.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:31:05.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:31:05.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:31:05.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:31:05.223 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:31:05.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:31:05.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:31:05.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:31:05.239 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:31:05.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:05.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:05.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:31:05.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:31:05.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:31:05.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:31:05.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:31:05.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:31:05.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:31:05.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:31:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:31:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:31:05.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:31:05.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:31:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:31:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:31:05.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:31:05.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:31:05.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:31:05.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:31:05.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:31:05.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:31:05.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:31:05.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:31:05.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:31:05.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:31:05.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:31:05.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:31:05.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:31:05.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:31:05.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:31:05.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:31:05.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:05.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:05.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:31:05.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:31:05.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:31:05.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:31:05.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:31:05.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:05.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:05.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:31:05.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:31:05.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:31:05.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-29T06:31:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:31:05.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:31:05.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:31:05.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:31:05.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:31:05.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T06:31:05.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:31:05.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:31:05.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:31:05.682 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:31:05.682 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:05.682 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:05.682 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:31:05.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T06:31:05.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:31:05.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:31:05.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:31:05.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:31:05.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:31:05.922 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:31:05.922 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:31:06.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:31:06.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:31:06.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:31:06.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:31:06.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:31:06.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:31:06.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:31:06.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-29T06:31:06.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:31:06.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:31:06.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:31:06.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:31:06.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:31:06.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:31:07.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:31:07.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:31:07.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:31:07.060 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:31:07.060 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:07.060 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:07.061 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:31:07.061 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:31:07.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:31:07.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-29T06:31:07.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-29T06:31:07.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-29T06:31:07.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:31:07.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:31:07.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:31:07.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:31:07.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:31:07.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:31:07.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:31:07.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:31:07.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:31:07.103 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:31:07.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2bcd0f59-d68a-4a98-a181-968133277095' 2024-03-29T06:31:07.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:31:07.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArYAZmUS5HBxAAaohHgi1kkvBaVMAKFXnqTw== 2024-03-29T06:31:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArYAZmUS5HBxAAaohHgi1kkvBaVMAKFXnqTw=="}' 2024-03-29T06:31:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2bcd0f59-d68a-4a98-a181-968133277095 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:31:07.403 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:31:07.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:31:07.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArYAZmUS5HBxAAaohHgi1kkvBaVMAKFXnqTw== --osd-uuid 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:31:07.451 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:07.452+0000 7f531e1907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:07.521 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:07.520+0000 7f531e1907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:07.523 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:07.524+0000 7f531e1907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:07.524 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:07.524+0000 7f531e1907c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:31:07.524 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:07.524+0000 7f531e1907c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:31:07.524 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:07.524+0000 7f531e1907c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:31:07.629 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:07.628+0000 7f531e1907c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:31:15.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:31:15.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:31:15.614 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:31:15.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:31:15.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:31:16.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:31:16.002 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:31:16.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:31:16.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:31:16.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:31:16.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:31:16.033 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:16.032+0000 7f42efa8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:16.035 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:16.036+0000 7f42efa8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:16.037 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:16.036+0000 7f42efa8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:16.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:31:16.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:31:16.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:31:16.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:31:16.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:31:16.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:16.300 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:31:16.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:31:16.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:16.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:16.605 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:16.604+0000 7f42efa8c7c0 -1 Falling back to public interface 2024-03-29T06:31:16.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:17.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:17.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:17.669 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:31:17.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:31:17.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:17.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:17.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:18.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:18.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:18.968 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:31:18.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:31:18.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:18.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:19.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:19.570 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:19.568+0000 7f42efa8c7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:31:20.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:20.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:20.279 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:31:20.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:31:20.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:20.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:20.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:21.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:21.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:21.728 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:31:21.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:31:21.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:21.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:22.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:23.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:23.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:23.036 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:31:23.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:31:23.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:23.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:23.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:24.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:24.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:24.433 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:31:24.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:31:24.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:24.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:24.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:25.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:25.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:25.942 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:31:25.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:31:25.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:25.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:26.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:27.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:27.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:27.256 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:31:27.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:31:27.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:27.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:27.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:28.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:28.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:28.766 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:31:28.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:31:28.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:28.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:29.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:30.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:30.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:30.088 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:31:30.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:31:30.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:30.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:30.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:31.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:31.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:31.423 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:31:31.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:31:31.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:31.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:32.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:33.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:33.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:33.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:31:33.073 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:31:33.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:33.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:33.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:34.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:31:34.471 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:31:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:34.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:35.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:35.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:35.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:31:35.887 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:31:35.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:35.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:36.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:37.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:37.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:37.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:31:37.245 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:31:37.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:37.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:37.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:31:38.745 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:31:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:39.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:40.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:40.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:40.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:31:40.057 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:31:40.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:40.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:40.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:41.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:41.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:41.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:31:41.381 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:31:41.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:41.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:41.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:42.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:42.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:42.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:31:42.970 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:31:42.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:42.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:43.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:44.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:44.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:44.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:31:44.883 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:31:44.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:44.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:45.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:46.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:46.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:46.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:31:46.512 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:31:46.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:46.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:47.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:48.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:48.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:48.341 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:31:48.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:31:48.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:48.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:48.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:49.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:49.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:49.682 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:31:49.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:31:49.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:49.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:50.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:51.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:51.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:51.578 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:31:51.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:31:51.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:51.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:52.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:31:53.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:31:53.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:31:53.046 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:31:53.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:31:53.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:31:53.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:31:53.356 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/469041299,v1:127.0.0.1:6803/469041299] [v2:127.0.0.1:6804/469041299,v1:127.0.0.1:6805/469041299] exists,up 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:31:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:31:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:31:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:31:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-29T06:31:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-29T06:31:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:31:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:31:53.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:31:53.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:31:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:31:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:31:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:31:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:31:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:31:53.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:31:53.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:31:53.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:31:53.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25' 2024-03-29T06:31:53.363 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:31:53.363 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:31:53.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZYAZmyCr1FhAAoC16Z9Vix7a9EggQzhur+Q== 2024-03-29T06:31:53.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZYAZmyCr1FhAAoC16Z9Vix7a9EggQzhur+Q=="}' 2024-03-29T06:31:53.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 -i td/osd-scrub-repair/1/new.json 2024-03-29T06:31:53.878 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:31:53.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:31:53.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZYAZmyCr1FhAAoC16Z9Vix7a9EggQzhur+Q== --osd-uuid 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:31:53.935 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:53.935+0000 7f0e3a1717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:54.011+0000 7f0e3a1717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:54.015+0000 7f0e3a1717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:31:54.016 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:54.015+0000 7f0e3a1717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:31:54.016 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:54.015+0000 7f0e3a1717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:31:54.016 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:54.015+0000 7f0e3a1717c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:31:54.111 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:31:54.111+0000 7f0e3a1717c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:32:02.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:32:02.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:32:02.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:32:02.574 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:32:02.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:32:02.964 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:32:02.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:32:02.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:32:02.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:32:02.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:32:02.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:32:02.996 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:02.995+0000 7fe867bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:02.999 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:02.995+0000 7fe867bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:03.000 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:02.999+0000 7fe867bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:03.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:32:03.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:32:03.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:32:03.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:32:03.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:32:03.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:03.275 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:32:03.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:32:03.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:03.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:03.574 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:03.571+0000 7fe867bfc7c0 -1 Falling back to public interface 2024-03-29T06:32:03.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:04.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:04.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:04.590 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:32:04.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:32:04.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:04.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:04.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:05.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:05.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:05.910 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:32:05.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:32:05.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:05.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:06.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:06.664 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:06.663+0000 7fe867bfc7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:32:07.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:07.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:07.240 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:32:07.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:32:07.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:07.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:07.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:08.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:08.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:08.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:32:08.714 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:32:08.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:08.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:09.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:10.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:10.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:10.034 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:32:10.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:32:10.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:10.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:10.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:11.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:11.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:11.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:32:11.456 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:32:11.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:11.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:12.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:13.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:13.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:13.014 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:32:13.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:32:13.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:13.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:13.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:14.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:14.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:14.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:32:14.335 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:32:14.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:14.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:14.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:15.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:15.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:15.796 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:32:15.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:32:15.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:15.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:16.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:17.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:17.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:17.180 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:32:17.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:32:17.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:17.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:17.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:18.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:18.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:18.636 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:32:18.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:32:18.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:18.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:19.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:20.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:20.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:20.125 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:32:20.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:32:20.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:20.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:20.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:21.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:21.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:21.449 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:32:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:32:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:21.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:21.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:22.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:22.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:22.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:32:22.853 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:32:22.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:22.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:23.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:24.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:24.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:24.156 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:32:24.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:32:24.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:24.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:24.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:25.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:25.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:25.497 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:32:25.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:32:25.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:25.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:26.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:27.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:27.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:27.004 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:32:27.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:32:27.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:27.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:27.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:28.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:28.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:28.321 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:32:28.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:32:28.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:28.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:29.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:30.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:30.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:30.276 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:32:30.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:32:30.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:30.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:30.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:31.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:31.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:31.979 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:32:31.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:32:31.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:31.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:32.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:33.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:33.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:33.734 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:32:33.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:32:33.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:33.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:34.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:35.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:35.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:35.777 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:32:35.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:32:35.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:35.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:36.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:37.087 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:32:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:32:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:37.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:38.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:38.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:38.747 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:32:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:32:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:38.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:39.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:40.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:40.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:40.300 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:32:40.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:32:40.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:40.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:40.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:41.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:41.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:41.782 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:32:41.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:32:41.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:41.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:32:42.174 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3487122028,v1:127.0.0.1:6811/3487122028] [v2:127.0.0.1:6812/3487122028,v1:127.0.0.1:6813/3487122028] exists,up 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:32:42.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:32:42.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:32:42.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:32:42.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:32:42.181 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:32:42.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=40444618-c14e-4b2c-bf19-edea14c65e46 2024-03-29T06:32:42.181 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 40444618-c14e-4b2c-bf19-edea14c65e46 2024-03-29T06:32:42.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 40444618-c14e-4b2c-bf19-edea14c65e46' 2024-03-29T06:32:42.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:32:42.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKYAZmoHElDBAAVadhmXg3NAcodk1VRR9vQw== 2024-03-29T06:32:42.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKYAZmoHElDBAAVadhmXg3NAcodk1VRR9vQw=="}' 2024-03-29T06:32:42.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 40444618-c14e-4b2c-bf19-edea14c65e46 -i td/osd-scrub-repair/2/new.json 2024-03-29T06:32:42.613 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:32:42.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T06:32:42.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKYAZmoHElDBAAVadhmXg3NAcodk1VRR9vQw== --osd-uuid 40444618-c14e-4b2c-bf19-edea14c65e46 2024-03-29T06:32:42.669 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:42.670+0000 7fededd627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:42.765 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:42.766+0000 7fededd627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:42.767 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:42.770+0000 7fededd627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:42.767 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:42.770+0000 7fededd627c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:32:42.768 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:42.770+0000 7fededd627c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:32:42.768 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:42.770+0000 7fededd627c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:32:42.876 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:42.878+0000 7fededd627c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T06:32:51.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T06:32:51.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:32:51.242 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T06:32:51.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T06:32:51.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:32:51.616 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T06:32:51.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T06:32:51.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:32:51.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:32:51.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:32:51.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:32:51.649 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:51.650+0000 7f90bc08a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:51.694 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:51.694+0000 7f90bc08a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:51.696 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:51.698+0000 7f90bc08a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:32:51.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T06:32:51.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:32:51.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:32:51.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:32:51.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:32:51.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:51.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:32:51.935 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:32:51.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:51.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:32:52.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:52.304 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:52.306+0000 7f90bc08a7c0 -1 Falling back to public interface 2024-03-29T06:32:53.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:53.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:53.256 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:32:53.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:32:53.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:53.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:32:53.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:54.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:54.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:54.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:32:54.559 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:32:54.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:54.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:32:54.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:55.331 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:32:55.330+0000 7f90bc08a7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T06:32:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:32:55.881 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:32:55.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:55.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:32:56.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:57.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:57.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:57.362 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:32:57.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:32:57.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:57.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:32:57.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:32:58.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:32:58.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:32:58.684 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:32:58.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:32:58.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:32:58.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:32:59.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:00.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:00.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:00.040 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:33:00.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:33:00.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:00.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:00.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:01.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:01.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:01.572 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:33:01.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:33:01.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:01.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:01.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:02.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:02.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:02.889 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:33:02.890 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:33:02.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:02.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:03.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:04.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:04.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:04.213 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:33:04.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:33:04.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:04.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:04.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:05.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:05.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:05.583 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:33:05.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:33:05.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:05.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:05.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:06.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:06.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:06.955 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:33:06.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:33:06.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:06.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:07.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:08.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:08.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:08.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:33:08.222 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:33:08.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:08.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:09.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:09.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:09.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:33:09.622 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:33:09.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:09.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:10.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:11.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:11.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:11.087 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:33:11.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:33:11.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:11.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:11.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:12.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:12.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:12.467 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:33:12.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:33:12.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:12.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:12.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:13.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:13.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:13.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:33:13.891 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:33:13.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:13.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:14.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:15.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:15.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:15.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:33:15.214 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:33:15.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:15.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:15.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:16.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:16.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:16.583 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:33:16.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:33:16.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:16.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:17.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:18.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:18.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:18.160 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:33:18.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:33:18.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:18.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:18.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:19.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:19.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:19.924 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:33:19.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:33:19.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:19.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:20.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:21.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:21.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:21.664 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:33:21.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:33:21.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:21.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:22.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:23.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:23.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:23.397 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:33:23.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:33:23.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:23.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:24.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:25.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:25.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:25.448 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:33:25.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:33:25.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:25.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:25.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:26.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:26.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:26.814 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:33:26.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:33:26.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:26.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:27.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:28.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:28.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:28.448 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:33:28.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:33:28.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:28.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:28.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:29.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:29.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:29.763 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:33:29.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:33:29.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:29.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:33:30.099 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2833547175,v1:127.0.0.1:6819/2833547175] [v2:127.0.0.1:6820/2833547175,v1:127.0.0.1:6821/2833547175] exists,up 40444618-c14e-4b2c-bf19-edea14c65e46 2024-03-29T06:33:30.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:33:30.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:33:30.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:33:30.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-29T06:33:30.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:33:30.408 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:33:30.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:33:30.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:33:30.872 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:33:30.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:33:31.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:33:35.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-29T06:33:35.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-29T06:33:35.646 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'foo' created 2024-03-29T06:33:35.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:33:36.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-03-29T06:33:36.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T06:33:36.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T06:33:36.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-29T06:33:36.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T06:33:36.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-29T06:33:37.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T06:33:37.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T06:33:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T06:33:38.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:33:39.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-29T06:33:39.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T06:33:39.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:33:39.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:33:39.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:33:39.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:33:39.551 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:33:39.551 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:33:39.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:33:39.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:33:39.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:33:39.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:33:39.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:33:39.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:33:39.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:33:39.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:33:39.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:33:39.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:33:39.995 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:33:39.995 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:33:39.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:33:39.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:39.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:33:40.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-29T06:33:40.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-29T06:33:40.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-29T06:33:40.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:40.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:33:40.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-29T06:33:40.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-29T06:33:40.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672981' 2024-03-29T06:33:40.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:40.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:33:40.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-29T06:33:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-29T06:33:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672981 2-64424509451' 2024-03-29T06:33:40.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:33:40.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-29T06:33:40.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:33:40.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:33:40.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-29T06:33:40.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:33:40.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-29T06:33:40.481 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836509 2024-03-29T06:33:40.481 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-29T06:33:40.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:33:40.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-29T06:33:40.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:33:41.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:33:41.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:33:42.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-29T06:33:42.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:33:42.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-29T06:33:42.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:33:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:33:42.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-29T06:33:42.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:33:42.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-29T06:33:42.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-29T06:33:42.388 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672981 2024-03-29T06:33:42.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:33:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672981 2024-03-29T06:33:42.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:33:42.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-29T06:33:42.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:33:42.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:33:42.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-29T06:33:42.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:33:42.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-29T06:33:42.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-29T06:33:42.825 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509451 2024-03-29T06:33:42.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:33:43.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-03-29T06:33:43.320 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:33:43.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:33:43.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:33:43.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:33:43.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:33:43.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:33:43.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:33:43.990 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:33:44.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:33:44.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:33:44.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:33:44.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T06:33:44.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-29T06:33:44.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:33:44.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:33:44.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:33:44.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:33:44.333 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:33:44.333 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:33:44.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:33:44.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:33:44.334 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:33:44.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:33:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:33:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:33:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:33:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:33:44.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:33:44.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:33:44.725 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:33:44.725 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:33:44.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:33:44.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:44.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:33:44.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-03-29T06:33:44.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-03-29T06:33:44.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-03-29T06:33:44.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:44.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:33:45.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-29T06:33:45.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-29T06:33:45.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672983' 2024-03-29T06:33:45.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:45.061 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:33:45.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-29T06:33:45.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-29T06:33:45.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672983 2-64424509453' 2024-03-29T06:33:45.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:33:45.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-03-29T06:33:45.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:33:45.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:33:45.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:33:45.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-03-29T06:33:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-03-29T06:33:45.203 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836511 2024-03-29T06:33:45.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-03-29T06:33:45.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:33:45.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-03-29T06:33:45.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:33:46.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:33:46.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:33:46.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-03-29T06:33:46.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:33:46.822 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-29T06:33:46.822 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:33:46.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:33:46.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-29T06:33:46.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:33:46.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-29T06:33:46.826 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672983 2024-03-29T06:33:46.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-29T06:33:46.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:33:47.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-29T06:33:47.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:33:47.137 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-29T06:33:47.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:33:47.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:33:47.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-29T06:33:47.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:33:47.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-29T06:33:47.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-29T06:33:47.142 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509453 2024-03-29T06:33:47.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:33:47.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-29T06:33:47.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:33:47.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:33:47.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:33:47.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:33:47.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:33:47.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:33:47.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:33:47.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:33:47.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:33:47.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:33:47.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:33:48.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:33:48.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:33:48.165 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:33:48.165 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:33:48.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:33:48.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:33:48.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:33:48.560 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-29T06:33:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:33:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-29T06:33:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-29T06:33:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:33:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:33:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-29T06:33:48.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:33:48.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:33:48.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:33:49.019 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:33:49.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:33:49.437 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:33:49.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:33:49.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:33:49.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:33:49.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-29T06:33:49.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:33:49.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:33:49.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:33:49.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:33:49.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:33:49.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:33:49.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:33:49.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:33:51.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:33:51.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:33:51.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:33:51.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:33:51.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:33:51.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:33:51.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:33:51.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:33:54.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:33:54.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:33:54.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:33:54.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:33:54.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:33:54.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:33:54.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:33:54.193 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:33:54.193 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:33:54.193 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:33:54.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:33:54.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:33:54.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:33:54.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:33:54.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:33:54.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:33:54.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:33:54.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:33:54.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:33:54.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:33:54.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:33:54.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:33:54.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:33:54.196 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:33:54.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:33:54.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:33:54.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:33:54.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:33:54.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:33:54.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:33:54.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:33:54.228+0000 7f8bdbcac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:33:54.233 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:33:54.232+0000 7f8bdbcac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:33:54.234 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:33:54.232+0000 7f8bdbcac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:33:54.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:54.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:33:54.769 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:33:54.768+0000 7f8bdbcac7c0 -1 Falling back to public interface 2024-03-29T06:33:54.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:55.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:55.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:55.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:33:55.831 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:33:55.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:55.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:33:56.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:57.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:57.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:57.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:33:57.146 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:33:57.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:57.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:33:57.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:33:57.851 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:33:57.848+0000 7f8bdbcac7c0 -1 osd.1 34 log_to_monitors true 2024-03-29T06:33:58.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:33:58.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:33:58.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:33:58.467 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:33:58.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:33:58.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 39 up_thru 29 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/2599090003,v1:127.0.0.1:6811/2599090003] [v2:127.0.0.1:6812/2599090003,v1:127.0.0.1:6813/2599090003] exists,up 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:33:58.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:33:58.797 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:33:58.798 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:33:58.798 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:33:58.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:33:58.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:33:58.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:33:58.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:33:58.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:33:58.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:33:58.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:33:58.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:33:58.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:33:59.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:33:59.526 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:33:59.526 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:33:59.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:33:59.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:59.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:33:59.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836515 2024-03-29T06:33:59.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836515 2024-03-29T06:33:59.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515' 2024-03-29T06:33:59.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:33:59.925 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:34:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-29T06:34:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-29T06:34:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-167503724546' 2024-03-29T06:34:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:00.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:34:00.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-29T06:34:00.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-29T06:34:00.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836515 1-167503724546 2-64424509457' 2024-03-29T06:34:00.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:00.528 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836515 2024-03-29T06:34:00.528 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:00.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:34:00.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836515 2024-03-29T06:34:00.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:00.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836515 2024-03-29T06:34:00.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-03-29T06:34:00.532 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836515 2024-03-29T06:34:00.532 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:34:00.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836515 2024-03-29T06:34:00.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:34:01.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:34:01.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:34:02.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836515 2024-03-29T06:34:02.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:02.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:02.183 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-03-29T06:34:02.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:34:02.185 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-03-29T06:34:02.185 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-29T06:34:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-03-29T06:34:02.186 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 167503724546 2024-03-29T06:34:02.186 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:34:02.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-29T06:34:02.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:02.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-29T06:34:02.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:02.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:34:02.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-29T06:34:02.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:02.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-29T06:34:02.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-29T06:34:02.494 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509457 2024-03-29T06:34:02.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:34:02.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-29T06:34:02.793 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:34:02.793 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:02.793 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:03.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:34:03.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:03.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:03.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:03.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:03.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:03.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:03.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:03.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:34:03.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:03.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:03.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:03.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:34:03.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-29T06:34:03.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:34:03.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:34:03.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:34:03.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:34:03.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:34:03.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:03.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:03.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:03.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:03.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:03.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:03.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:34:04.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:04.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:04.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:34:04.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:34:05.065 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:34:05.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:34:05.592 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:34:05.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:34:05.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:34:05.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:34:05.915 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-29T06:34:05.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-29T06:34:05.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:34:05.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:1+0 records in 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:1+0 records out 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000159386 s, 12.8 MB/s 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:34:05.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:34:05.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:34:05.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:34:05.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:34:05.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:34:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:34:05.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:34:07.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:34:07.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:34:07.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:34:07.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:34:07.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:34:07.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:34:07.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:34:07.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:34:10.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:34:10.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:34:10.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:34:10.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:34:10.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:34:10.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:34:10.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:34:10.077 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:34:10.077 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:34:10.077 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:34:10.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:34:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:34:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:34:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:34:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:34:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:34:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:34:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:34:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:34:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:34:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:34:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:34:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:34:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:34:10.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:34:10.080 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:34:10.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:34:10.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:34:10.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:34:10.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:34:10.113 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:10.112+0000 7f52fd49e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:10.116 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:10.116+0000 7f52fd49e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:10.117 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:10.116+0000 7f52fd49e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:10.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:10.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:10.682 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:10.680+0000 7f52fd49e7c0 -1 Falling back to public interface 2024-03-29T06:34:10.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:11.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:11.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:11.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:34:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:34:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:12.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:13.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:13.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:13.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:34:13.131 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:34:13.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:13.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:13.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:13.655 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:13.652+0000 7f52fd49e7c0 -1 osd.0 43 log_to_monitors true 2024-03-29T06:34:14.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:14.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:14.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:34:14.453 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:34:14.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:14.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 48 up_thru 39 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/3599365182,v1:127.0.0.1:6803/3599365182] [v2:127.0.0.1:6804/3599365182,v1:127.0.0.1:6805/3599365182] exists,up 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:34:14.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:34:14.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:34:14.988 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:34:14.988 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:34:14.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:34:14.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:34:14.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:34:15.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:34:15.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:34:15.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:34:15.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:34:15.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:34:15.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:34:15.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:34:15.603 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:34:15.603 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:34:15.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:34:15.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:15.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:34:15.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-03-29T06:34:15.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-03-29T06:34:15.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-03-29T06:34:15.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:15.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:34:16.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724550 2024-03-29T06:34:16.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724550 2024-03-29T06:34:16.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550' 2024-03-29T06:34:16.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:16.548 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:34:16.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-29T06:34:16.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-29T06:34:16.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550 2-64424509461' 2024-03-29T06:34:16.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:16.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-03-29T06:34:16.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:16.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:34:16.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-03-29T06:34:16.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:16.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-03-29T06:34:16.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-03-29T06:34:16.730 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 206158430210 2024-03-29T06:34:16.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:34:17.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-03-29T06:34:17.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:34:18.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:34:18.152 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:34:18.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-03-29T06:34:18.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:18.522 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724550 2024-03-29T06:34:18.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:18.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:34:18.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724550 2024-03-29T06:34:18.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:18.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724550 2024-03-29T06:34:18.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2024-03-29T06:34:18.526 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 167503724550 2024-03-29T06:34:18.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:34:18.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724550 -lt 167503724550 2024-03-29T06:34:18.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:18.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:18.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-29T06:34:18.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:34:18.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-29T06:34:18.886 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:18.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-29T06:34:18.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-29T06:34:18.887 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509461 2024-03-29T06:34:18.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:34:19.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-29T06:34:19.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:34:19.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:19.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:19.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:34:19.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:19.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:19.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:19.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:19.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:19.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:19.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:19.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:34:19.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:19.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:19.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:20.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:34:20.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:34:20.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:34:20.770 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:34:20.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:34:21.363 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:34:21.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:34:21.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:34:21.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:34:21.870 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-29T06:34:21.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:34:21.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:34:21.874 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:34:21.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:34:21.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:34:21.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:34:23.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:34:23.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-29T06:34:23.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:34:23.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:34:23.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:34:23.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:34:23.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:34:23.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-29T06:34:25.203 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-29T06:34:25.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:34:25.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:34:25.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:34:25.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:34:25.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:34:25.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:34:25.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:34:25.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:34:25.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:34:25.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:34:25.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:34:25.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:34:25.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:34:25.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:34:25.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:34:25.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:34:25.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:34:25.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:34:25.746 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:34:25.746 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:34:25.746 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:34:25.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:34:25.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:34:25.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:34:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:34:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:34:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:34:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:34:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:34:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:34:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:34:25.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:34:25.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:34:25.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:34:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:34:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:34:25.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:34:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:34:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:34:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:34:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:34:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:25.780+0000 7f34aa9cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:25.794 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:25.792+0000 7f34aa9cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:25.795 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:25.796+0000 7f34aa9cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:26.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:34:26.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:34:26.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:26.329 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:26.327+0000 7f34aa9cf7c0 -1 Falling back to public interface 2024-03-29T06:34:26.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:27.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:27.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:27.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:34:27.451 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:34:27.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:27.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:34:27.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:28.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:28.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:28.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:34:28.765 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:34:28.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:28.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:34:29.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:29.208 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:29.183+0000 7f34aa9cf7c0 -1 osd.1 52 log_to_monitors true 2024-03-29T06:34:30.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:30.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:30.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:34:30.083 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:34:30.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:30.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/2740471034,v1:127.0.0.1:6811/2740471034] [v2:127.0.0.1:6812/2740471034,v1:127.0.0.1:6813/2740471034] exists,up 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:34:30.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:34:30.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:34:30.483 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:34:30.483 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:34:30.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:34:30.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:34:30.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:34:30.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:34:30.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:34:30.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:34:30.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:34:30.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:34:30.599 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:34:31.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:34:31.078 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:34:31.078 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:34:31.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:34:31.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:31.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:34:31.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430214 2024-03-29T06:34:31.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430214 2024-03-29T06:34:31.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214' 2024-03-29T06:34:31.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:31.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:34:31.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-03-29T06:34:31.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-03-29T06:34:31.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-244813135874' 2024-03-29T06:34:31.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:31.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:34:31.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-29T06:34:31.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-29T06:34:31.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-244813135874 2-64424509465' 2024-03-29T06:34:31.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:31.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430214 2024-03-29T06:34:31.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:31.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:34:31.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430214 2024-03-29T06:34:31.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:31.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430214 2024-03-29T06:34:31.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430214' 2024-03-29T06:34:31.764 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 206158430214 2024-03-29T06:34:31.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:34:32.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430214 2024-03-29T06:34:32.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:32.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-03-29T06:34:32.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:32.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:34:32.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-03-29T06:34:32.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:32.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-03-29T06:34:32.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-03-29T06:34:32.212 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135874 2024-03-29T06:34:32.212 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:34:32.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-03-29T06:34:32.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:32.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-29T06:34:32.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:32.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:34:32.608 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-29T06:34:32.608 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:32.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-29T06:34:32.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-29T06:34:32.610 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509465 2024-03-29T06:34:32.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:34:32.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509465 2024-03-29T06:34:32.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:34:33.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:34:33.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:34:34.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-03-29T06:34:34.324 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:34:34.325 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:34.325 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:34.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:34.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:34:34.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:34.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:34.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:35.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:34:35.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:34:35.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:34:35.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:34:35.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:34:35.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:34:35.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:34:35.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:35.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:35.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:35.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:35.442 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:35.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:35.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:35.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:34:35.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:35.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:35.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:34:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:34:36.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-29T06:34:36.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:34:36.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:34:36.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:34:36.590 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:34:36.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:34:37.066 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:34:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:34:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:34:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:34:37.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-29T06:34:37.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-29T06:34:37.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:34:37.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-29T06:34:37.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-29T06:34:38.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:34:38.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:34:38.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:34:38.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:34:38.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:34:38.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:34:38.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:34:38.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:34:39.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:34:39.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:34:39.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:34:39.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:34:39.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:34:39.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:34:39.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:34:39.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:34:42.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:34:42.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:34:42.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:34:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:34:42.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:34:42.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:34:42.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:34:42.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:34:42.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:34:42.198 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:34:42.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:34:42.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:34:42.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:34:42.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:34:42.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:34:42.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:34:42.232 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:42.231+0000 7f688cef77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:42.236 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:42.235+0000 7f688cef77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:42.237 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:42.235+0000 7f688cef77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:34:42.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:34:42.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:34:42.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:34:42.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:34:42.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:34:42.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:42.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:34:42.519 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:34:42.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:42.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:42.786 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:42.783+0000 7f688cef77c0 -1 Falling back to public interface 2024-03-29T06:34:42.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:43.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:43.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:43.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:34:43.831 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:34:43.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:43.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:44.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:45.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:45.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:45.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:34:45.145 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:34:45.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:45.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:45.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:34:45.688 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:34:45.687+0000 7f688cef77c0 -1 osd.0 61 log_to_monitors true 2024-03-29T06:34:46.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:34:46.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:34:46.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:34:46.458 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:34:46.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:34:46.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:34:46.768 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 66 up_thru 58 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/3291007957,v1:127.0.0.1:6803/3291007957] [v2:127.0.0.1:6804/3291007957,v1:127.0.0.1:6805/3291007957] exists,up 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:34:46.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:34:46.770 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:34:46.770 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:34:46.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:34:46.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:34:46.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:34:46.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:34:46.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:34:46.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:34:46.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:34:46.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:34:46.869 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:34:47.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:34:47.250 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:34:47.250 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:34:47.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:34:47.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:47.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:34:47.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-29T06:34:47.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-29T06:34:47.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-03-29T06:34:47.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:47.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:34:48.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-03-29T06:34:48.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-03-29T06:34:48.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135878' 2024-03-29T06:34:48.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:34:48.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:34:48.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509469 2024-03-29T06:34:48.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509469 2024-03-29T06:34:48.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135878 2-64424509469' 2024-03-29T06:34:48.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:48.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-03-29T06:34:48.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:48.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:34:48.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-03-29T06:34:48.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:48.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-29T06:34:48.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-03-29T06:34:48.290 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 283467841538 2024-03-29T06:34:48.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:34:48.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-03-29T06:34:48.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:48.953 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135878 2024-03-29T06:34:48.953 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:48.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:34:48.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135878 2024-03-29T06:34:48.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:48.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-03-29T06:34:48.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-03-29T06:34:48.958 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135878 2024-03-29T06:34:48.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:34:49.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135877 -lt 244813135878 2024-03-29T06:34:49.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:34:50.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:34:50.326 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:34:50.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135879 -lt 244813135878 2024-03-29T06:34:50.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:34:50.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509469 2024-03-29T06:34:50.649 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:34:50.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:34:50.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509469 2024-03-29T06:34:50.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:34:50.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509469 2024-03-29T06:34:50.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-03-29T06:34:50.653 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509469 2024-03-29T06:34:50.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:34:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509470 -lt 64424509469 2024-03-29T06:34:51.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:34:51.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:51.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:51.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:34:51.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:51.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:51.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:51.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:51.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:51.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:51.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:51.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:34:51.762 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:51.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:51.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:34:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:34:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:34:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:34:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:34:52.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:34:52.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:34:52.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:52.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:52.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:52.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:52.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:52.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:52.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:52.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:34:52.505 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:52.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:52.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:52.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:34:52.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:34:52.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:34:52.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:34:52.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:34:52.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:34:52.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:34:52.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:34:52.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:34:53.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:34:53.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:34:53.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:34:53.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:34:53.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:34:53.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:34:53.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:53.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:53.460 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:53.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:53.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:53.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:53.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:53.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:34:53.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:53.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:53.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:54.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:34:54.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:34:54.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:34:54.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:34:54.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:34:54.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:34:54.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:34:54.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:34:54.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:34:54.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:34:54.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:34:54.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:34:54.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:34:54.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:34:54.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:34:54.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:54.893 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:54.893 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:54.893 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:54.893 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:54.894 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:54.894 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:55.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:34:55.187 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:55.187 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:55.188 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:34:55.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:34:55.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:34:55.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:34:55.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:34:55.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-29T06:34:55.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:34:55.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-29T06:34:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:34:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:34:56.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:34:56.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:34:56.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:34:56.700 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:34:56.700 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:34:56.700 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:34:57.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:34:57.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:34:57.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:34:57.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:34:57.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:34:57.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:34:57.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:34:57.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:34:57.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:34:57.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:34:57.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:34:57.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:34:57.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:34:58.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:34:58.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:34:58.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:34:58.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:34:58.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:34:58.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:34:58.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:34:58.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:35:01.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:35:01.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:35:01.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:35:01.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:35:01.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:35:01.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:35:01.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:35:01.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:35:01.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:35:01.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:35:01.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:35:01.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:35:01.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:35:01.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:35:01.305 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.2 2024-03-29T06:35:01.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:35:01.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:35:01.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:35:01.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:35:01.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:35:01.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:35:01.336 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:01.335+0000 7f38816777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:01.339 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:01.339+0000 7f38816777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:01.340 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:01.339+0000 7f38816777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:35:01.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:01.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:01.881 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:01.879+0000 7f38816777c0 -1 Falling back to public interface 2024-03-29T06:35:01.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:02.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:02.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:02.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:35:02.923 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:35:02.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:02.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:03.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:04.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:04.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:04.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:35:04.238 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:35:04.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:04.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:04.628 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:04.626+0000 7f38816777c0 -1 osd.2 69 log_to_monitors true 2024-03-29T06:35:04.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:05.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:05.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:05.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:35:05.656 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:35:05.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:05.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:osd.2 up in weight 1 up_from 74 up_thru 67 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/4053173096,v1:127.0.0.1:6819/4053173096] [v2:127.0.0.1:6820/4053173096,v1:127.0.0.1:6821/4053173096] exists,up 40444618-c14e-4b2c-bf19-edea14c65e46 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:35:06.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:35:06.082 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:35:06.083 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:35:06.083 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:35:06.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:35:06.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:35:06.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:35:06.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:35:06.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:35:06.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:35:06.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:35:06.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:35:06.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:35:06.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:35:06.732 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:35:06.732 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:35:06.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:35:06.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:06.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:35:07.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841543 2024-03-29T06:35:07.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841543 2024-03-29T06:35:07.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841543' 2024-03-29T06:35:07.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:07.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:35:07.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135883 2024-03-29T06:35:07.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135883 2024-03-29T06:35:07.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841543 1-244813135883' 2024-03-29T06:35:07.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:07.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:35:07.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-29T06:35:07.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-29T06:35:07.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841543 1-244813135883 2-317827579906' 2024-03-29T06:35:07.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:07.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841543 2024-03-29T06:35:07.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:07.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:35:07.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841543 2024-03-29T06:35:07.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:07.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841543 2024-03-29T06:35:07.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841543' 2024-03-29T06:35:07.553 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 283467841543 2024-03-29T06:35:07.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:35:07.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841543 2024-03-29T06:35:07.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:35:08.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:35:08.928 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:35:09.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841543 2024-03-29T06:35:09.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:09.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135883 2024-03-29T06:35:09.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:09.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:35:09.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135883 2024-03-29T06:35:09.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:09.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135883 2024-03-29T06:35:09.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135883' 2024-03-29T06:35:09.255 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135883 2024-03-29T06:35:09.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:35:09.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135883 -lt 244813135883 2024-03-29T06:35:09.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:09.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-29T06:35:09.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:09.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:35:09.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-29T06:35:09.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:09.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-29T06:35:09.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-29T06:35:09.595 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 317827579906 2024-03-29T06:35:09.595 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:35:09.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-29T06:35:09.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:35:09.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:09.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:10.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:10.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:35:10.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:10.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:10.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:10.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:35:10.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-29T06:35:10.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:35:10.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:35:10.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:10.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:35:11.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:11.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:11.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:11.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:11.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:11.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:11.089 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:11.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:11.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:35:11.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:11.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:11.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:11.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:35:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-29T06:35:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:35:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:35:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:35:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:35:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:35:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:35:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:35:12.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:35:12.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:35:12.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:35:12.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:12.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:35:12.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:12.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:12.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:12.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:12.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:12.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:12.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:12.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:12.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:35:12.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:12.682 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:12.682 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:35:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:35:13.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:35:13.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:35:13.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:35:13.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:35:13.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:35:13.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:35:13.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:35:13.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:35:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:35:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:35:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:35:14.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:35:14.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:35:14.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:35:14.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:35:14.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:35:14.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:35:14.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:35:14.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:35:16.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:35:16.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:35:16.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:35:16.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:35:16.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:35:16.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:35:16.849 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.2 2024-03-29T06:35:16.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:35:16.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:35:16.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:35:16.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:35:16.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:35:16.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:35:16.882 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:16.882+0000 7f9c35e9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:16.884 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:16.886+0000 7f9c35e9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:16.886 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:16.886+0000 7f9c35e9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:35:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:35:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:35:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:35:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:35:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:35:17.172 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:35:17.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:17.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:17.422 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:17.422+0000 7f9c35e9f7c0 -1 Falling back to public interface 2024-03-29T06:35:17.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:18.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:18.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:18.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:35:18.529 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:35:18.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:18.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:18.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:19.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:19.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:19.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:35:19.831 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:35:19.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:19.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:20.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:20.439 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:20.438+0000 7f9c35e9f7c0 -1 osd.2 76 log_to_monitors true 2024-03-29T06:35:21.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:21.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:21.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:35:21.150 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:35:21.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:21.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:35:21.630 INFO:tasks.workunit.client.0.smithi123.stderr:osd.2 up in weight 1 up_from 81 up_thru 74 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/3686099048,v1:127.0.0.1:6819/3686099048] [v2:127.0.0.1:6820/3686099048,v1:127.0.0.1:6821/3686099048] exists,up 40444618-c14e-4b2c-bf19-edea14c65e46 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:35:21.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:35:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:35:21.632 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:35:21.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:35:21.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:35:21.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:35:21.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:35:21.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:35:21.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:35:21.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:35:21.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:35:21.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:35:22.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:35:22.196 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:35:22.196 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:35:22.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:35:22.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:22.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:35:22.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841547 2024-03-29T06:35:22.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841547 2024-03-29T06:35:22.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547' 2024-03-29T06:35:22.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:22.490 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:35:22.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135887 2024-03-29T06:35:22.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135887 2024-03-29T06:35:22.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547 1-244813135887' 2024-03-29T06:35:22.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:22.765 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:35:22.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-03-29T06:35:22.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-03-29T06:35:22.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547 1-244813135887 2-347892350978' 2024-03-29T06:35:22.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:22.915 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841547 2024-03-29T06:35:22.915 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:22.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:35:22.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841547 2024-03-29T06:35:22.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:22.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841547 2024-03-29T06:35:22.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841547' 2024-03-29T06:35:22.918 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 283467841547 2024-03-29T06:35:22.918 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:35:23.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841546 -lt 283467841547 2024-03-29T06:35:23.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:35:24.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:35:24.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:35:24.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841547 -lt 283467841547 2024-03-29T06:35:24.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:24.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:24.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135887 2024-03-29T06:35:24.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:35:24.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:24.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135887 2024-03-29T06:35:24.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135887 2024-03-29T06:35:24.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135887' 2024-03-29T06:35:24.618 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135887 2024-03-29T06:35:24.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:35:24.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135887 -lt 244813135887 2024-03-29T06:35:24.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:24.951 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350978 2024-03-29T06:35:24.951 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:24.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:35:24.953 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350978 2024-03-29T06:35:24.953 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:24.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-03-29T06:35:24.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-03-29T06:35:24.955 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 347892350978 2024-03-29T06:35:24.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:35:25.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-03-29T06:35:25.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:35:25.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:25.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:25.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:35:25.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:25.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:25.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:25.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:25.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:25.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:25.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:26.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:35:26.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:26.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:26.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:26.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:35:26.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-29T06:35:26.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:35:26.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:35:26.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:26.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:35:26.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:26.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:26.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:26.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:26.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:26.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:26.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:26.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:26.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:35:26.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:26.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:26.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:35:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-29T06:35:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:35:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:35:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:35:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:35:27.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:35:27.157 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:35:27.157 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:35:27.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:35:27.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:35:27.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:35:27.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:27.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:35:27.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:27.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:27.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:27.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:27.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:27.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:27.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:27.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:28.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:35:28.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:28.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:28.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:28.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:35:28.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:35:28.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:35:28.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-29T06:35:28.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:35:28.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-29T06:35:28.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-29T06:35:28.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:35:28.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:35:28.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-29T06:35:28.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:35:28.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:35:28.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:35:28.826 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:35:28.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:35:29.394 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:35:29.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:35:29.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:35:29.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:35:29.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-29T06:35:29.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:35:29.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:35:29.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-29T06:35:29.772 INFO:tasks.workunit.client.0.smithi123.stderr:2+0 records in 2024-03-29T06:35:29.772 INFO:tasks.workunit.client.0.smithi123.stderr:2+0 records out 2024-03-29T06:35:29.772 INFO:tasks.workunit.client.0.smithi123.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000173127 s, 23.7 MB/s 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:35:29.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:35:29.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:35:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:35:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:35:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:35:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:35:31.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:35:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:35:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:35:31.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:35:33.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:35:33.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:35:33.966 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:35:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:35:33.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:35:33.968 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:35:33.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:35:33.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:35:33.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:35:33.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:35:33.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:35:33.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:35:34.001 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:34.002+0000 7fd0baa8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:34.004 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:34.006+0000 7fd0baa8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:34.005 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:34.006+0000 7fd0baa8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:34.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:35:34.538 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:34.538+0000 7fd0baa8f7c0 -1 Falling back to public interface 2024-03-29T06:35:34.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:35.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:35.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:35.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:35:35.620 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:35:35.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:35.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:35:35.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:36.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:36.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:36.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:35:36.941 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:35:36.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:36.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:35:37.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:37.440 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:37.442+0000 7fd0baa8f7c0 -1 osd.1 85 log_to_monitors true 2024-03-29T06:35:38.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:38.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:38.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:35:38.247 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:35:38.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:38.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:35:38.976 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 90 up_thru 81 down_at 86 last_clean_interval [57,85) [v2:127.0.0.1:6810/828919922,v1:127.0.0.1:6811/828919922] [v2:127.0.0.1:6812/828919922,v1:127.0.0.1:6813/828919922] exists,up 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:35:38.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:35:38.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:35:38.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:35:38.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:35:38.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:35:38.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:35:38.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:35:38.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:35:38.977 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:35:38.977 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:35:38.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:35:38.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:35:38.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:35:39.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:35:39.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:35:39.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:35:39.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:35:39.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:35:39.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:35:39.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:35:39.518 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:35:39.518 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:35:39.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:35:39.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:39.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:35:39.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841551 2024-03-29T06:35:39.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841551 2024-03-29T06:35:39.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841551' 2024-03-29T06:35:39.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:39.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:35:40.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-03-29T06:35:40.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-03-29T06:35:40.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841551 1-386547056642' 2024-03-29T06:35:40.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:35:40.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:35:40.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350982 2024-03-29T06:35:40.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350982 2024-03-29T06:35:40.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841551 1-386547056642 2-347892350982' 2024-03-29T06:35:40.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:40.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841551 2024-03-29T06:35:40.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:40.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:35:40.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841551 2024-03-29T06:35:40.356 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:40.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841551 2024-03-29T06:35:40.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841551' 2024-03-29T06:35:40.357 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 283467841551 2024-03-29T06:35:40.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:35:40.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841550 -lt 283467841551 2024-03-29T06:35:40.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:35:41.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:35:41.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:35:42.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841552 -lt 283467841551 2024-03-29T06:35:42.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:42.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-03-29T06:35:42.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:42.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:35:42.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-03-29T06:35:42.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:42.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-03-29T06:35:42.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-03-29T06:35:42.023 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 386547056642 2024-03-29T06:35:42.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:35:42.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-03-29T06:35:42.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:35:42.424 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350982 2024-03-29T06:35:42.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:35:42.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:35:42.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350982 2024-03-29T06:35:42.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:35:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350982 2024-03-29T06:35:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350982' 2024-03-29T06:35:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 347892350982 2024-03-29T06:35:42.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:35:42.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350983 -lt 347892350982 2024-03-29T06:35:42.738 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:35:42.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:42.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:43.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:35:43.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:43.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:43.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:43.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:43.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:43.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:43.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:43.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:35:43.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:43.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:43.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:43.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:35:43.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:35:43.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:35:43.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:35:43.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:43.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:35:43.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:43.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:43.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:43.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:43.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:43.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:43.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:43.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:44.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:35:44.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:44.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:44.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:44.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:35:44.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:35:44.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:35:44.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:35:44.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:35:44.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:35:44.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:35:44.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:35:44.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:35:45.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:35:45.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:35:45.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:35:45.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:45.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:35:45.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:45.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:45.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:45.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:45.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:45.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:45.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:45.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:45.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:35:45.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:45.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:45.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:45.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:35:45.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:35:45.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:35:45.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:35:45.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:35:45.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:35:45.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:35:45.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:35:45.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:35:46.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:35:46.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:35:46.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:35:46.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:46.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:35:46.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:46.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:46.757 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:46.757 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:46.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:46.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:46.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:46.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:47.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:35:47.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:47.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:47.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:47.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:35:47.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:35:47.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:35:47.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:35:47.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:35:47.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:35:47.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:35:47.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:35:47.412 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:35:47.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:35:47.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:35:47.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-29T06:35:47.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:35:47.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-29T06:35:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:35:48.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:35:48.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:35:48.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:35:48.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:35:48.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:35:48.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:35:48.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:35:48.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:35:48.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:35:48.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:35:48.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:35:49.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:35:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:35:49.803 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:35:49.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:35:50.388 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:35:50.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:35:50.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:35:50.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:35:50.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-29T06:35:50.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-29T06:35:50.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:35:50.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:35:50.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:35:50.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:35:50.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:35:51.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:35:51.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:35:51.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:35:51.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:35:51.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:35:51.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:35:51.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:35:51.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:35:54.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:35:54.697 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:35:54.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:35:54.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:35:54.700 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:35:54.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:35:54.701 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:35:54.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:35:54.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:35:54.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:35:54.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:35:54.733 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:54.733+0000 7faf9741f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:54.736 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:54.737+0000 7faf9741f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:54.738 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:54.737+0000 7faf9741f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:35:55.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:35:55.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:35:55.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:35:55.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:35:55.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:35:55.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:55.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:35:55.018 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:35:55.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:55.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:35:55.281 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:55.281+0000 7faf9741f7c0 -1 Falling back to public interface 2024-03-29T06:35:55.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:56.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:56.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:56.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:35:56.315 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:35:56.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:56.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:35:56.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:57.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:57.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:57.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:35:57.625 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:35:57.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:57.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:35:57.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:35:58.281 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:35:58.281+0000 7faf9741f7c0 -1 osd.0 94 log_to_monitors true 2024-03-29T06:35:58.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:35:58.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:35:58.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:35:58.933 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:35:58.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:35:58.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:35:59.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:00.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:00.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:00.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:36:00.279 INFO:tasks.workunit.client.0.smithi123.stderr:4 2024-03-29T06:36:00.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:00.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:00.815 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 99 up_thru 99 down_at 95 last_clean_interval [66,94) [v2:127.0.0.1:6802/2300631898,v1:127.0.0.1:6803/2300631898] [v2:127.0.0.1:6804/2300631898,v1:127.0.0.1:6805/2300631898] exists,up 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:36:00.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:36:00.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:36:00.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:36:00.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:36:00.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:36:00.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:36:00.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:36:00.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:36:00.817 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:36:00.817 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:36:00.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:36:00.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:36:00.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:36:00.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:36:00.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:36:00.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:36:00.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:36:00.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:36:00.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:36:01.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:36:01.419 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:36:01.419 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:36:01.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:36:01.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:01.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:36:01.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-29T06:36:01.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-29T06:36:01.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306' 2024-03-29T06:36:01.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:01.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:36:01.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056647 2024-03-29T06:36:01.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056647 2024-03-29T06:36:01.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056647' 2024-03-29T06:36:01.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:01.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:36:01.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350988 2024-03-29T06:36:01.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350988 2024-03-29T06:36:01.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056647 2-347892350988' 2024-03-29T06:36:01.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:01.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762306 2024-03-29T06:36:01.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:01.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:36:01.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762306 2024-03-29T06:36:01.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:01.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-29T06:36:01.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762306' 2024-03-29T06:36:01.863 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 425201762306 2024-03-29T06:36:01.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:36:02.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-29T06:36:02.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:02.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056647 2024-03-29T06:36:02.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:02.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:36:02.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:02.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056647 2024-03-29T06:36:02.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056647 2024-03-29T06:36:02.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056647' 2024-03-29T06:36:02.164 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 386547056647 2024-03-29T06:36:02.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:36:02.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056647 -lt 386547056647 2024-03-29T06:36:02.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:02.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350988 2024-03-29T06:36:02.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:02.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:36:02.543 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350988 2024-03-29T06:36:02.543 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:02.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350988 2024-03-29T06:36:02.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350988' 2024-03-29T06:36:02.545 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 347892350988 2024-03-29T06:36:02.545 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:36:02.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350987 -lt 347892350988 2024-03-29T06:36:02.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:36:03.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:36:03.898 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:36:04.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350988 -lt 347892350988 2024-03-29T06:36:04.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:36:04.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:04.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:04.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:36:04.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:04.641 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:04.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:04.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:04.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:04.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:04.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:04.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:36:04.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:04.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:04.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:05.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:36:05.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:36:05.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:36:05.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-29T06:36:05.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:36:05.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:36:05.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:36:05.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:36:05.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:36:08.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:36:08.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:36:08.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:36:08.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:36:08.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:36:08.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:36:08.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:36:08.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:36:08.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:36:08.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:36:08.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:36:08.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:36:08.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:36:08.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:36:08.746 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:36:08.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:36:08.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:36:08.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:36:08.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:36:08.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:36:08.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:36:08.780 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:08.781+0000 7f5be2d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:08.783 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:08.781+0000 7f5be2d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:08.784 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:08.785+0000 7f5be2d0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:09.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:36:09.322 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:09.321+0000 7f5be2d0e7c0 -1 Falling back to public interface 2024-03-29T06:36:09.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:10.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:10.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:10.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:36:10.430 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:36:10.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:10.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:36:10.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:11.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:11.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:11.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:36:11.752 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:36:11.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:11.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:36:12.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:12.433 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:12.433+0000 7f5be2d0e7c0 -1 osd.1 101 log_to_monitors true 2024-03-29T06:36:13.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:13.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:13.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:36:13.079 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:36:13.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:13.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:36:13.482 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 106 up_thru 99 down_at 102 last_clean_interval [90,101) [v2:127.0.0.1:6810/2143069168,v1:127.0.0.1:6811/2143069168] [v2:127.0.0.1:6812/2143069168,v1:127.0.0.1:6813/2143069168] exists,up 51d28ad6-5fcd-47f5-b1ef-b1fea6828a25 2024-03-29T06:36:13.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:36:13.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:36:13.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:36:13.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:36:13.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:36:13.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:36:13.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:36:13.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:36:13.484 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:36:13.484 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:36:13.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:36:13.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:36:13.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:36:13.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:36:13.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:36:13.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:36:13.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:36:13.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:36:13.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:36:14.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:36:14.254 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:36:14.254 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:36:14.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:36:14.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:14.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:36:14.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762310 2024-03-29T06:36:14.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762310 2024-03-29T06:36:14.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762310' 2024-03-29T06:36:14.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:14.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:36:14.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-03-29T06:36:14.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-03-29T06:36:14.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762310 1-455266533378' 2024-03-29T06:36:14.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:14.877 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:36:15.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350991 2024-03-29T06:36:15.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350991 2024-03-29T06:36:15.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762310 1-455266533378 2-347892350991' 2024-03-29T06:36:15.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:15.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762310 2024-03-29T06:36:15.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:15.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:36:15.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762310 2024-03-29T06:36:15.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:15.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762310 2024-03-29T06:36:15.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762310' 2024-03-29T06:36:15.030 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 425201762310 2024-03-29T06:36:15.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:36:15.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762308 -lt 425201762310 2024-03-29T06:36:15.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:36:16.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:36:16.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:36:16.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762310 -lt 425201762310 2024-03-29T06:36:16.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:16.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-03-29T06:36:16.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:16.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:36:16.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-03-29T06:36:16.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:16.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-03-29T06:36:16.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-03-29T06:36:16.840 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 455266533378 2024-03-29T06:36:16.841 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:36:17.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-03-29T06:36:17.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:17.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350991 2024-03-29T06:36:17.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:17.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:36:17.170 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:17.170 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350991 2024-03-29T06:36:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350991 2024-03-29T06:36:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350991' 2024-03-29T06:36:17.171 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 347892350991 2024-03-29T06:36:17.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:36:17.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350991 -lt 347892350991 2024-03-29T06:36:17.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:36:17.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:17.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:17.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:36:17.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:17.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:17.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:17.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:17.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:17.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:17.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:18.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:36:18.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:18.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:18.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:36:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-29T06:36:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:36:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:36:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:36:18.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:36:18.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:36:18.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:18.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:18.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:18.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:18.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:18.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:18.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:19.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:36:19.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:19.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:19.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:19.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:36:19.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-29T06:36:19.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:36:19.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:36:19.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:36:19.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:36:19.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:36:19.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:36:19.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:36:19.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:36:19.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:36:19.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:36:19.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:36:19.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:36:20.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:36:20.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:20.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:20.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:20.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:20.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:20.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:20.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:20.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:36:20.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:20.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:20.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:20.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:36:20.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:36:20.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:36:20.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:36:21.378 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:36:21.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:36:21.829 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:36:21.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:36:21.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:36:21.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:36:22.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-29T06:36:22.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:36:22.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:36:22.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-29T06:36:22.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-29T06:36:22.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-29T06:36:22.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-29T06:36:22.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:36:22.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:36:22.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:36:22.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:36:22.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:36:22.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:36:22.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:36:22.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:36:22.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:36:25.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:36:25.571 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:36:25.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:36:25.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:36:25.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:36:25.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:36:25.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:36:25.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:36:25.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:36:25.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:36:25.607 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:25.604+0000 7f1fd026e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:25.609 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:25.608+0000 7f1fd026e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:25.611 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:25.608+0000 7f1fd026e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:25.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:36:25.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:36:25.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:36:25.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:36:25.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:36:25.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:25.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:36:25.903 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:36:25.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:25.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:26.154 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:26.152+0000 7f1fd026e7c0 -1 Falling back to public interface 2024-03-29T06:36:26.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:27.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:27.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:27.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:36:27.262 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:36:27.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:27.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:27.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:28.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:28.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:28.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:36:28.580 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:36:28.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:28.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:28.832 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:28.832+0000 7f1fd026e7c0 -1 osd.0 110 log_to_monitors true 2024-03-29T06:36:28.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:29.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:29.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:29.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:36:29.905 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:36:29.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:29.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [99,110) [v2:127.0.0.1:6802/4285068726,v1:127.0.0.1:6803/4285068726] [v2:127.0.0.1:6804/4285068726,v1:127.0.0.1:6805/4285068726] exists,up 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:36:30.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:36:30.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:36:30.287 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:36:30.287 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:36:30.288 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:36:30.288 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:36:30.288 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:36:30.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:36:30.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:36:30.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:36:30.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:36:30.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:36:30.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:36:31.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:36:31.054 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:36:31.054 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:36:31.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:36:31.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:31.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:36:31.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-03-29T06:36:31.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-03-29T06:36:31.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042' 2024-03-29T06:36:31.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:31.327 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:36:31.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533382 2024-03-29T06:36:31.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533382 2024-03-29T06:36:31.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533382' 2024-03-29T06:36:31.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:31.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:36:31.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350996 2024-03-29T06:36:31.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350996 2024-03-29T06:36:31.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533382 2-347892350996' 2024-03-29T06:36:31.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:31.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:31.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-493921239042 2024-03-29T06:36:31.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:36:31.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-493921239042 2024-03-29T06:36:31.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:31.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-03-29T06:36:31.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-03-29T06:36:31.617 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 493921239042 2024-03-29T06:36:31.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:36:31.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 493921239042 2024-03-29T06:36:31.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:36:32.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:36:32.933 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:36:33.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239042 -lt 493921239042 2024-03-29T06:36:33.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:33.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533382 2024-03-29T06:36:33.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:33.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:36:33.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533382 2024-03-29T06:36:33.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:33.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533382 2024-03-29T06:36:33.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533382' 2024-03-29T06:36:33.254 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 455266533382 2024-03-29T06:36:33.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:36:33.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533382 -lt 455266533382 2024-03-29T06:36:33.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:33.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:33.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350996 2024-03-29T06:36:33.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:36:33.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350996 2024-03-29T06:36:33.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:33.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350996 2024-03-29T06:36:33.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350996' 2024-03-29T06:36:33.576 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 347892350996 2024-03-29T06:36:33.576 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:36:33.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350996 -lt 347892350996 2024-03-29T06:36:33.905 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:36:33.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:33.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:34.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:36:34.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:34.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:34.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:34.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:34.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:34.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:34.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:34.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:36:34.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:34.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:34.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:35.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:36:35.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:36:35.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:36:35.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:36:35.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:36:35.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:36:35.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:36:35.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:35.152 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:35.152 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:35.152 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:35.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:35.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:35.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:35.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:36:35.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:35.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:35.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:35.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:36:35.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:36:36.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:36:36.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:36:36.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:36:36.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:36:36.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:36:36.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:36:36.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:36.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:36.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:36.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:36.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:36.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:36.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:36.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:36:36.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:36.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:36.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:37.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:36:37.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:36:37.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:36:37.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:36:37.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:36:37.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:36:37.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:36:37.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:36:37.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:36:37.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:36:37.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:36:37.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:36:37.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:36:37.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:36:37.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:36:37.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:37.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:37.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:37.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:37.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:37.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:37.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:38.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:36:38.177 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:38.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:38.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:36:38.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:36:38.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:36:38.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:36:42.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:36:42.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:36:42.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:36:42.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:36:42.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:36:42.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:36:42.280 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:36:42.280 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:36:42.280 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:36:42.280 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:36:42.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:36:42.281 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:36:42.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:36:42.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:36:42.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:36:42.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:36:42.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:36:42.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:36:42.314 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:42.312+0000 7eff6ff4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:42.323 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:42.320+0000 7eff6ff4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:42.324 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:42.324+0000 7eff6ff4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:42.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:42.866 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:42.864+0000 7eff6ff4a7c0 -1 Falling back to public interface 2024-03-29T06:36:42.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:43.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:43.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:43.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:36:43.974 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:36:43.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:43.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:44.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:45.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:45.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:45.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:36:45.292 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:36:45.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:45.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:45.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:36:45.740 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:36:45.720+0000 7eff6ff4a7c0 -1 osd.0 117 log_to_monitors true 2024-03-29T06:36:46.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:36:46.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:36:46.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:36:46.614 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:36:46.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:36:46.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:36:47.218 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 122 up_thru 115 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6802/2863895499,v1:127.0.0.1:6803/2863895499] [v2:127.0.0.1:6804/2863895499,v1:127.0.0.1:6805/2863895499] exists,up 2bcd0f59-d68a-4a98-a181-968133277095 2024-03-29T06:36:47.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:36:47.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:36:47.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:36:47.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:36:47.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:36:47.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:36:47.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:36:47.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:36:47.220 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:36:47.220 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:36:47.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:36:47.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:36:47.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:36:47.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:36:47.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:36:47.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:36:47.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:36:47.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:36:47.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:36:47.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:36:47.930 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:36:47.930 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:36:47.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:36:47.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:47.930 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:36:48.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-03-29T06:36:48.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-03-29T06:36:48.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-03-29T06:36:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:48.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:36:48.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533387 2024-03-29T06:36:48.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533387 2024-03-29T06:36:48.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533387' 2024-03-29T06:36:48.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:36:48.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:36:48.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892351000 2024-03-29T06:36:48.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892351000 2024-03-29T06:36:48.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533387 2-347892351000' 2024-03-29T06:36:48.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:48.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-03-29T06:36:48.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:48.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:36:48.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-03-29T06:36:48.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:48.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-03-29T06:36:48.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-03-29T06:36:48.870 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 523986010114 2024-03-29T06:36:48.870 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:36:49.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-03-29T06:36:49.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:36:50.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:36:50.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:36:50.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-03-29T06:36:50.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:50.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533387 2024-03-29T06:36:50.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:50.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:36:50.559 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533387 2024-03-29T06:36:50.559 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:50.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533387 2024-03-29T06:36:50.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533387' 2024-03-29T06:36:50.560 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 455266533387 2024-03-29T06:36:50.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:36:50.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533387 -lt 455266533387 2024-03-29T06:36:50.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:36:50.928 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892351000 2024-03-29T06:36:50.929 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:36:50.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:36:50.932 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892351000 2024-03-29T06:36:50.932 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:36:50.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892351000 2024-03-29T06:36:50.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892351000' 2024-03-29T06:36:50.933 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 347892351000 2024-03-29T06:36:50.934 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:36:51.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892351000 -lt 347892351000 2024-03-29T06:36:51.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:36:51.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:51.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:51.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:36:51.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:36:51.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:36:51.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:36:51.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:36:51.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:36:51.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:36:51.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:36:51.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:36:51.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:36:51.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:36:51.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:36:52.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:36:52.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:36:52.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:36:52.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-29T06:36:52.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-29T06:36:52.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:36:52.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-29T06:36:52.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-29T06:36:52.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:36:52.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-29T06:36:52.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-29T06:36:52.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-29T06:36:52.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-29T06:36:52.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:36:52.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:36:52.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:36:52.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:36:52.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:52.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-29T06:36:53.217 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-29T06:36:53.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:53.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:36:53.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:53.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:36:53.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:36:53.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:36:53.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:36:53.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:36:53.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:36:53.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:36:53.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:36:53.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:33:38.208278+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:53.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:36:54.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:36:54.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:36:54.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:36:54.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:36:54.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:36:54.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:36:54.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:36:54.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:33:38.208278+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:54.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:36:55.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:36:55.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:36:55.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:36:55.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:36:55.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:36:55.850 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:36:55.850 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:36:56.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:33:38.208278+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:56.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:36:57.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:36:57.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:36:57.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:36:57.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:36:57.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:36:57.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:36:57.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:36:57.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:33:38.208278+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:57.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:36:58.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:36:58.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:36:58.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:36:58.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:36:58.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:36:58.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:36:58.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:36:58.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:33:38.208278+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:36:58.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:36:59.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:36:59.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:36:59.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:36:59.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:36:59.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:36:59.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:36:59.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:37:00.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:36:53.643172+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:37:00.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:37:00.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-29T06:37:00.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-29T06:37:00.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-29T06:37:00.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-29T06:37:00.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-29T06:37:00.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-29T06:37:00.258 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-29T06:37:00.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-03-29T06:37:00.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T06:37:00.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:37:00.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:37:00.328 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-29T06:37:00.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:37:00.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:37:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:37:00.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:37:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-29T06:37:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-29T06:37:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-29T06:37:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-29T06:37:00.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:37:00.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:37:00.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:37:00.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:37:00.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:37:00.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-29T06:33:38.208278+0000 2024-03-29T06:37:00.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-29T06:37:00.944 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-29T06:37:00.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-29T06:33:38.208278+0000 last_deep_scrub_stamp 2024-03-29T06:37:00.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:37:00.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:33:38.208278+0000 2024-03-29T06:37:00.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-29T06:37:00.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:37:00.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:37:00.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:37:00.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:37:00.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:37:00.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:37:00.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:37:01.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:33:38.208278+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:37:01.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:37:02.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:37:02.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:37:02.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:37:02.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:37:02.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:37:02.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:37:02.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:37:02.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:33:38.208278+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:37:02.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:37:03.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:37:03.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:37:03.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:37:03.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:37:03.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:37:03.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:37:03.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:37:03.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:37:01.722043+0000 '>' 2024-03-29T06:33:38.208278+0000 2024-03-29T06:37:03.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:37:03.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-29T06:37:03.925 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-29T06:37:03.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-29T06:37:03.953 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-29T06:37:03.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-29T06:37:03.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-29T06:37:04.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-29T06:37:04.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:37:04.039 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:37:04.040 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:37:04.040 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T06:37:04.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:37:04.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:37:04.083 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:37:04.084 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:37:04.084 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-29T06:37:04.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:37:04.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:37:04.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:37:04.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:37:04.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-29T06:37:04.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-29T06:37:04.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-29T06:37:04.522 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' removed 2024-03-29T06:37:04.541 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:37:04.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:37:04.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:37:04.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:37:04.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:37:04.542 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:37:04.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:37:04.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:37:04.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:37:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:37:04.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:37:04.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:37:04.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:37:04.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:37:04.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:37:04.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:37:04.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:37:04.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:37:04.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:37:04.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:37:04.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:37:04.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:37:04.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:37:04.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:37:04.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:04.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:04.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:37:04.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:37:04.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:37:04.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:37:04.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:37:04.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:37:04.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:37:04.908 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:37:04.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:37:04.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:37:04.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:37:04.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:37:04.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:37:04.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:37:04.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:37:04.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:37:04.914 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:37:04.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:37:04.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:37:04.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:37:04.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:37:04.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:04.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:04.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:37:04.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:37:04.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:37:04.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:37:04.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:37:04.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:04.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:04.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:37:04.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:37:04.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:37:04.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:37:04.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:37:04.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T06:37:05.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:37:05.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:37:05.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:37:05.375 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:37:05.375 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:05.375 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:05.376 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:37:05.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T06:37:05.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:37:05.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:37:05.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:37:05.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:37:05.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:37:05.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:37:05.631 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:37:05.631 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:37:05.631 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:37:05.632 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:37:05.632 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:05.632 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:05.632 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:37:05.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:37:05.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:37:05.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:37:05.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:37:05.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:37:05.750 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:37:05.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:37:05.752 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:37:05.752 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:37:05.753 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:37:05.753 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:37:05.753 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:05.753 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:05.754 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:37:05.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:37:05.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:37:05.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-29T06:37:05.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:37:05.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:37:05.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:37:05.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:37:05.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:37:05.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:37:07.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:37:07.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:37:07.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:37:07.553 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:37:07.553 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:07.553 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:07.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:37:07.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:37:07.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:37:07.590 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:37:07.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:07.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:37:07.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:37:07.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:37:07.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:37:07.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:37:07.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4068ae4d-160b-4d7d-b085-8dd763ecada0' 2024-03-29T06:37:07.597 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:37:07.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:37:07.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTYQZmpMWwJBAAsswe8caUguW3vsBKXgMcdQ== 2024-03-29T06:37:07.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTYQZmpMWwJBAAsswe8caUguW3vsBKXgMcdQ=="}' 2024-03-29T06:37:07.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4068ae4d-160b-4d7d-b085-8dd763ecada0 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:37:07.880 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:37:07.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:37:07.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTYQZmpMWwJBAAsswe8caUguW3vsBKXgMcdQ== --osd-uuid 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:37:07.927 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:07.923+0000 7f909001b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:07.964 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:07.963+0000 7f909001b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:07.965 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:07.963+0000 7f909001b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:07.965 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:07.963+0000 7f909001b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:37:07.966 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:07.963+0000 7f909001b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:37:07.966 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:07.963+0000 7f909001b7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:37:08.064 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:08.063+0000 7f909001b7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:37:16.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:37:16.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:37:16.490 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:37:16.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:37:16.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:37:16.879 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:37:16.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:37:16.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:37:16.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:37:16.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:37:16.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:37:16.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:16.911+0000 7f2ffa1087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:16.961 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:16.959+0000 7f2ffa1087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:16.962 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:16.959+0000 7f2ffa1087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:37:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:37:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:37:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:37:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:37:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:17.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:37:17.184 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:37:17.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:17.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:17.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:17.554 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:17.551+0000 7f2ffa1087c0 -1 Falling back to public interface 2024-03-29T06:37:18.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:18.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:18.497 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:37:18.497 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:37:18.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:18.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:18.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:19.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:19.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:19.816 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:37:19.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:37:19.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:19.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:20.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:20.491 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:20.487+0000 7f2ffa1087c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:37:21.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:21.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:21.137 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:37:21.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:37:21.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:21.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:21.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:22.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:22.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:22.517 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:37:22.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:37:22.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:22.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:22.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:23.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:23.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:23.833 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:37:23.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:37:23.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:23.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:24.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:25.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:25.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:25.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:37:25.136 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:37:25.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:25.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:25.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:26.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:26.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:26.608 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:37:26.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:37:26.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:26.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:26.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:27.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:27.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:27.913 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:37:27.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:37:27.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:27.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:28.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:29.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:29.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:29.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:37:29.226 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:37:29.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:29.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:29.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:30.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:30.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:30.696 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:37:30.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:37:30.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:30.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:31.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:32.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:32.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:32.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:37:32.023 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:37:32.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:32.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:32.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:33.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:33.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:33.469 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:37:33.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:37:33.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:33.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:33.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:34.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:34.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:34.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:37:34.876 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:37:34.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:34.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:35.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:36.306 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:37:36.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:37:36.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:36.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:36.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:37.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:37.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:37.726 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:37:37.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:37:37.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:37.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:38.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:39.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:39.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:39.049 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:37:39.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:37:39.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:39.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:39.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:40.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:40.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:40.386 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:37:40.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:37:40.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:40.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:40.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:41.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:41.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:41.700 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:37:41.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:37:41.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:41.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:42.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:43.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:43.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:43.418 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:37:43.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:37:43.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:43.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:44.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:45.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:45.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:45.384 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:37:45.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:37:45.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:45.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:45.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:46.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:46.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:46.884 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:37:46.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:37:46.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:46.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:47.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:48.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:48.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:48.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:37:48.633 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:37:48.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:48.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:49.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:50.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:50.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:50.321 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:37:50.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:37:50.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:50.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:51.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:52.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:52.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:52.368 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:37:52.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:37:52.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:52.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:52.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:37:53.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:37:53.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:37:53.691 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:37:53.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:37:53.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:37:53.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:37:54.008 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2206401177,v1:127.0.0.1:6803/2206401177] [v2:127.0.0.1:6804/2206401177,v1:127.0.0.1:6805/2206401177] exists,up 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:37:54.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:37:54.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:37:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:37:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:37:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:37:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:37:54.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:37:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:37:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 50ac9ad8-fdf4-4271-948a-7b35bd3f9150' 2024-03-29T06:37:54.016 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:37:54.016 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:37:54.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCYQZmRiM8AhAAknl4SENuno9rdfivK8VnQw== 2024-03-29T06:37:54.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCYQZmRiM8AhAAknl4SENuno9rdfivK8VnQw=="}' 2024-03-29T06:37:54.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 -i td/osd-scrub-repair/1/new.json 2024-03-29T06:37:54.433 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:37:54.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:37:54.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCYQZmRiM8AhAAknl4SENuno9rdfivK8VnQw== --osd-uuid 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:37:54.487 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:54.486+0000 7f183a1257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:54.561 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:54.562+0000 7f183a1257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:54.563 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:54.562+0000 7f183a1257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:37:54.563 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:54.566+0000 7f183a1257c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:37:54.564 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:54.566+0000 7f183a1257c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:37:54.564 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:54.566+0000 7f183a1257c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:37:54.644 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:37:54.646+0000 7f183a1257c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:38:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:38:02.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:38:02.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:38:02.878 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:38:02.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:38:03.256 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:38:03.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:38:03.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:38:03.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:38:03.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:38:03.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:38:03.289 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:03.290+0000 7f29925507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:03.339 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:03.338+0000 7f29925507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:03.341 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:03.342+0000 7f29925507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:03.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:38:03.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:38:03.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:38:03.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:38:03.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:38:03.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:03.564 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:38:03.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:38:03.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:03.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:03.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:03.950 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:03.950+0000 7f29925507c0 -1 Falling back to public interface 2024-03-29T06:38:04.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:04.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:04.904 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:38:04.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:38:04.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:04.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:05.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:06.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:06.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:06.223 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:38:06.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:38:06.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:06.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:06.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:07.078 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:07.078+0000 7f29925507c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:38:07.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:07.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:07.526 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:38:07.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:38:07.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:07.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:08.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:09.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:09.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:09.039 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:38:09.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:38:09.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:09.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:09.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:10.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:10.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:10.536 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:38:10.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:38:10.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:10.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:10.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:11.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:11.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:11.935 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:38:11.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:38:11.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:11.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:12.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:13.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:13.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:13.240 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:38:13.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:38:13.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:13.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:13.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:14.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:14.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:14.768 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:38:14.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:38:14.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:14.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:15.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:16.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:16.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:16.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:38:16.175 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:38:16.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:16.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:16.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:17.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:17.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:17.783 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:38:17.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:38:17.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:17.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:18.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:19.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:19.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:19.197 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:38:19.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:38:19.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:19.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:19.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:20.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:20.696 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:38:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:38:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:20.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:21.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:22.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:22.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:22.168 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:38:22.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:38:22.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:22.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:22.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:23.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:23.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:23.885 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:38:23.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:38:23.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:23.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:24.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:25.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:25.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:25.243 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:38:25.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:38:25.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:25.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:25.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:26.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:26.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:26.693 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:38:26.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:38:26.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:26.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:27.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:28.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:28.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:28.149 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:38:28.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:38:28.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:28.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:29.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:30.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:30.038 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:30.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:38:30.039 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:38:30.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:30.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:31.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:32.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:32.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:32.178 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:38:32.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:38:32.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:32.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:33.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:34.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:34.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:34.223 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:38:34.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:38:34.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:34.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:34.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:35.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:35.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:35.545 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:38:35.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:38:35.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:35.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:36.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:37.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:37.087 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:38:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:38:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:37.088 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:37.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:38.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:38.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:38.638 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:38:38.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:38:38.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:38.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:38.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:39.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:39.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:39.948 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:38:39.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:38:39.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:39.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:40.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:41.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:41.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:41.594 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:38:41.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:38:41.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:41.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:38:41.904 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3800806622,v1:127.0.0.1:6811/3800806622] [v2:127.0.0.1:6812/3800806622,v1:127.0.0.1:6813/3800806622] exists,up 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T06:38:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:38:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:38:41.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:38:41.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:38:41.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47883a79-6a81-4db0-8169-4e454edea1fd 2024-03-29T06:38:41.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 47883a79-6a81-4db0-8169-4e454edea1fd' 2024-03-29T06:38:41.913 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 47883a79-6a81-4db0-8169-4e454edea1fd 2024-03-29T06:38:41.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:38:41.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxYQZm/UGqNxAArQ20H5dVIMoBIEKUeEfatA== 2024-03-29T06:38:41.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxYQZm/UGqNxAArQ20H5dVIMoBIEKUeEfatA=="}' 2024-03-29T06:38:41.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47883a79-6a81-4db0-8169-4e454edea1fd -i td/osd-scrub-repair/2/new.json 2024-03-29T06:38:42.477 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:38:42.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T06:38:42.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxYQZm/UGqNxAArQ20H5dVIMoBIEKUeEfatA== --osd-uuid 47883a79-6a81-4db0-8169-4e454edea1fd 2024-03-29T06:38:42.540 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:42.537+0000 7fed233dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:42.641 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:42.641+0000 7fed233dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:42.643 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:42.641+0000 7fed233dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:42.643 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:42.645+0000 7fed233dc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:38:42.644 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:42.645+0000 7fed233dc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:38:42.644 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:42.645+0000 7fed233dc7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:38:42.694 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:42.693+0000 7fed233dc7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T06:38:51.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T06:38:51.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:38:51.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T06:38:51.126 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T06:38:51.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:38:51.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T06:38:51.514 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T06:38:51.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:38:51.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:38:51.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:38:51.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:38:51.545 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:51.545+0000 7f896f9487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:51.592 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:51.593+0000 7f896f9487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:51.594 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:51.593+0000 7f896f9487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:38:51.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T06:38:51.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:38:51.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:38:51.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:38:51.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:38:51.808 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:38:51.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:51.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:38:51.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:51.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:38:52.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:52.186 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:52.185+0000 7f896f9487c0 -1 Falling back to public interface 2024-03-29T06:38:53.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:53.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:53.100 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:38:53.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:38:53.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:38:53.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:53.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:54.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:54.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:54.420 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:38:54.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:38:54.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:54.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:38:54.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:55.312 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:38:55.313+0000 7f896f9487c0 -1 osd.2 0 log_to_monitors true 2024-03-29T06:38:55.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:55.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:55.765 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:38:55.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:38:55.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:55.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:38:56.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:57.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:57.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:57.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:38:57.327 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:38:57.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:57.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:38:57.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:38:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:38:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:38:58.875 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:38:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:38:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:38:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:38:59.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:00.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:00.193 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:39:00.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:39:00.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:00.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:00.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:01.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:01.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:01.639 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:39:01.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:39:01.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:01.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:01.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:02.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:02.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:02.954 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:39:02.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:39:02.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:02.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:03.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:04.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:04.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:04.479 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:39:04.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:39:04.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:04.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:04.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:05.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:05.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:05.833 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:39:05.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:39:05.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:05.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:06.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:07.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:07.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:07.207 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:39:07.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:39:07.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:07.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:07.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:08.567 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:39:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:39:08.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:08.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:08.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:09.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:09.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:09.880 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:39:09.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:39:09.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:09.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:10.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:11.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:11.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:11.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:39:11.236 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:39:11.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:11.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:11.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:12.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:12.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:12.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:39:12.590 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:39:12.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:12.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:12.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:13.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:13.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:13.911 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:39:13.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:39:13.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:13.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:14.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:15.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:15.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:15.250 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:39:15.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:39:15.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:15.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:16.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:17.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:17.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:17.057 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:39:17.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:39:17.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:17.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:17.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:18.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:18.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:18.571 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:39:18.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:39:18.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:18.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:19.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:20.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:20.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:20.703 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:39:20.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:39:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:21.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:22.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:39:22.716 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:39:22.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:22.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:23.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:24.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:24.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:24.959 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:39:24.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:39:24.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:24.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:25.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:26.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:26.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:26.874 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:39:26.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:39:26.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:26.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:27.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:28.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:28.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:28.645 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:39:28.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:39:28.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:28.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:39:28.961 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/207632220,v1:127.0.0.1:6819/207632220] [v2:127.0.0.1:6820/207632220,v1:127.0.0.1:6821/207632220] exists,up 47883a79-6a81-4db0-8169-4e454edea1fd 2024-03-29T06:39:28.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:39:28.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:39:28.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:39:28.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-29T06:39:28.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:39:29.431 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:39:29.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:39:29.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:39:30.047 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:39:30.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:39:31.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:39:34.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-29T06:39:34.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-29T06:39:34.621 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'foo' created 2024-03-29T06:39:34.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:39:35.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-03-29T06:39:35.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T06:39:35.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T06:39:35.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-29T06:39:35.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T06:39:35.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-29T06:39:36.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T06:39:36.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T06:39:37.297 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T06:39:37.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:39:38.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-29T06:39:38.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-29T06:39:39.354 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 3 allow_ec_overwrites to true 2024-03-29T06:39:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T06:39:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:39:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:39:39.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:39:39.379 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:39:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:39:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:39:39.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:39:39.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:39:39.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:39:39.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:39:39.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:39:39.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:39:39.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:39:39.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:39:39.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:39:39.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:39:39.905 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:39:39.905 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:39:39.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:39:39.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:39:39.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:39:40.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-29T06:39:40.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-29T06:39:40.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-29T06:39:40.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:39:40.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:39:40.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-29T06:39:40.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-29T06:39:40.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672980' 2024-03-29T06:39:40.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:39:40.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:39:40.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-29T06:39:40.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-29T06:39:40.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672980 2-64424509447' 2024-03-29T06:39:40.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:39:40.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:39:40.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-29T06:39:40.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:39:40.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-29T06:39:40.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:39:40.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-29T06:39:40.344 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836506 2024-03-29T06:39:40.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-29T06:39:40.345 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:39:40.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-29T06:39:40.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:39:40.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-29T06:39:40.641 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:39:40.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:39:40.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-29T06:39:40.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:39:40.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-29T06:39:40.645 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672980 2024-03-29T06:39:40.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-29T06:39:40.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:39:40.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-03-29T06:39:40.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:39:41.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:39:41.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:39:42.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-03-29T06:39:42.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:39:43.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T06:39:43.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:39:43.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672980 2024-03-29T06:39:43.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:39:43.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:39:43.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-29T06:39:43.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:39:43.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-29T06:39:43.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:39:43.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-29T06:39:43.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-29T06:39:43.649 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509447 2024-03-29T06:39:43.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:39:43.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-29T06:39:43.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:39:43.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:39:43.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:39:44.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:39:44.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:39:44.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:39:44.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:39:44.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:39:44.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:39:44.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:39:44.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:39:44.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:39:44.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:39:44.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:39:44.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:39:44.998 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:39:44.999 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:39:44.999 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:39:44.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:39:44.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:39:44.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:39:45.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:39:45.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:39:45.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:39:45.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:39:45.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:39:45.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:39:45.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:39:45.396 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:39:45.396 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:39:45.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:39:45.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:39:45.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:39:45.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-29T06:39:45.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-29T06:39:45.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-29T06:39:45.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:39:45.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:39:45.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-29T06:39:45.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-29T06:39:45.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-03-29T06:39:45.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:39:45.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:39:45.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-29T06:39:45.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-29T06:39:45.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509449' 2024-03-29T06:39:45.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:39:45.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-29T06:39:45.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:39:45.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:39:45.848 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-29T06:39:45.848 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:39:45.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-29T06:39:45.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-29T06:39:45.850 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836508 2024-03-29T06:39:45.851 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:39:46.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-29T06:39:46.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:39:47.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:39:47.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:39:47.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-29T06:39:47.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:39:47.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-29T06:39:47.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:39:47.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:39:47.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:39:47.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-29T06:39:47.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-29T06:39:47.459 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672982 2024-03-29T06:39:47.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-29T06:39:47.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:39:47.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-03-29T06:39:47.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:39:47.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-29T06:39:47.752 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:39:47.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:39:47.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-29T06:39:47.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:39:47.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-29T06:39:47.757 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509449 2024-03-29T06:39:47.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-29T06:39:47.757 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:39:48.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-29T06:39:48.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:39:48.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:39:48.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:39:48.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:39:48.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:39:48.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:39:48.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:39:48.760 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:39:49.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:39:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:39:49.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:39:49.125 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:39:49.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:39:49.538 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:39:49.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:39:50.115 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:39:50.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:39:50.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:39:50.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:39:50.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-29T06:39:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:39:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:39:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-29T06:39:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-29T06:39:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:39:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:39:50.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:39:50.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:39:50.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:39:50.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:39:50.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:39:50.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:39:50.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:39:50.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:39:50.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:39:50.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:39:50.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:39:50.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:39:50.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:39:50.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:39:51.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:39:54.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:39:54.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:39:54.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:39:54.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:39:54.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:39:54.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:39:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:39:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:39:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:39:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:39:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:39:54.506 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:39:54.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:39:54.507 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:39:54.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:39:54.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:39:54.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:39:54.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:39:54.540 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:39:54.539+0000 7effe8ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:39:54.542 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:39:54.539+0000 7effe8ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:39:54.543 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:39:54.543+0000 7effe8ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:39:54.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:39:54.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:39:54.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:39:54.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:39:54.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:39:54.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:54.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:39:54.827 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:39:54.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:54.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:39:55.090 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:39:55.087+0000 7effe8ec47c0 -1 Falling back to public interface 2024-03-29T06:39:55.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:56.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:56.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:39:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:39:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:56.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:39:56.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:57.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:57.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:57.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:39:57.517 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:39:57.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:57.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:39:57.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:39:58.084 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:39:58.083+0000 7effe8ec47c0 -1 osd.1 35 log_to_monitors true 2024-03-29T06:39:58.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:39:58.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:39:58.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:39:58.827 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:39:58.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:39:58.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:39:59.180 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 39 up_thru 29 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/3654181449,v1:127.0.0.1:6811/3654181449] [v2:127.0.0.1:6812/3654181449,v1:127.0.0.1:6813/3654181449] exists,up 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:39:59.181 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:39:59.182 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:39:59.182 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:39:59.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:39:59.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:39:59.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:39:59.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:39:59.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:39:59.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:39:59.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:39:59.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:39:59.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:39:59.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:39:59.810 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:39:59.810 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:39:59.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:39:59.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:39:59.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:40:00.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-03-29T06:40:00.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-03-29T06:40:00.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-03-29T06:40:00.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:00.215 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:40:00.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-29T06:40:00.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-29T06:40:00.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-167503724546' 2024-03-29T06:40:00.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:00.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:40:00.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-29T06:40:00.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-29T06:40:00.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-167503724546 2-64424509453' 2024-03-29T06:40:00.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:00.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-03-29T06:40:00.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:00.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:40:00.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-03-29T06:40:00.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:00.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-03-29T06:40:00.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-03-29T06:40:00.829 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836512 2024-03-29T06:40:00.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:01.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836512 2024-03-29T06:40:01.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:40:02.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:40:02.141 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:02.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-03-29T06:40:02.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:02.469 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-03-29T06:40:02.469 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:02.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:40:02.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-03-29T06:40:02.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:02.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-29T06:40:02.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-03-29T06:40:02.472 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 167503724546 2024-03-29T06:40:02.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:40:02.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-29T06:40:02.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:02.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-29T06:40:02.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:02.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:40:02.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-29T06:40:02.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:02.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-29T06:40:02.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-29T06:40:02.771 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509453 2024-03-29T06:40:02.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:40:03.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-29T06:40:03.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:40:03.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:03.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:03.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:40:03.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:40:03.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:40:03.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:40:03.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:40:03.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:40:03.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:40:03.411 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:40:03.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:40:03.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:40:03.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:03.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:04.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:40:04.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-29T06:40:04.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:40:04.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:40:04.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:40:04.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:40:04.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:40:04.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:40:04.186 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:40:04.187 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:40:04.187 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:40:04.187 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:40:04.187 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:40:04.187 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:40:04.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:40:04.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:40:04.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:04.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:04.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:40:04.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:40:04.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:40:04.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:40:05.297 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:40:05.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:40:05.873 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:40:05.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:40:05.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:40:05.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:40:06.381 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-29T06:40:06.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-29T06:40:06.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:40:06.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-29T06:40:06.384 INFO:tasks.workunit.client.0.smithi123.stderr:1+0 records in 2024-03-29T06:40:06.384 INFO:tasks.workunit.client.0.smithi123.stderr:1+0 records out 2024-03-29T06:40:06.384 INFO:tasks.workunit.client.0.smithi123.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000281414 s, 7.3 MB/s 2024-03-29T06:40:06.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:40:06.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:40:06.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:40:06.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:40:06.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:40:06.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:40:06.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:40:06.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:40:07.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:40:07.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:40:07.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:40:07.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:40:07.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:40:07.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:40:07.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:40:07.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:40:10.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:40:10.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:40:10.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:40:10.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:40:10.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:40:10.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:40:10.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:40:10.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:40:10.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:40:10.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:40:10.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:40:10.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:40:10.549 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:40:10.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:40:10.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:40:10.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:40:10.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:40:10.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:40:10.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:40:10.581 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:10.579+0000 7fbc8fe247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:10.584 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:10.583+0000 7fbc8fe247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:10.586 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:10.583+0000 7fbc8fe247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:40:10.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:10.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:11.121 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:11.119+0000 7fbc8fe247c0 -1 Falling back to public interface 2024-03-29T06:40:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:12.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:12.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:12.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:40:12.198 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:40:12.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:12.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:12.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:13.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:13.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:13.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:40:13.514 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:40:13.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:13.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:13.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:14.028 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:14.027+0000 7fbc8fe247c0 -1 osd.0 43 log_to_monitors true 2024-03-29T06:40:14.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:14.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:14.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:40:14.835 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:40:14.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:14.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 48 up_thru 39 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/2254097604,v1:127.0.0.1:6803/2254097604] [v2:127.0.0.1:6804/2254097604,v1:127.0.0.1:6805/2254097604] exists,up 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:40:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:40:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:40:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:40:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:40:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:40:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:40:15.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:40:15.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:40:15.445 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:40:15.445 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:40:15.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:40:15.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:40:15.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:40:15.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:40:15.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:40:15.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:40:15.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:40:15.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:40:15.559 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:40:16.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:40:16.056 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:40:16.056 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:40:16.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:40:16.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:16.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:40:16.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-03-29T06:40:16.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-03-29T06:40:16.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-03-29T06:40:16.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:16.328 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:40:16.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724550 2024-03-29T06:40:16.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724550 2024-03-29T06:40:16.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550' 2024-03-29T06:40:16.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:16.692 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:40:16.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-29T06:40:16.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-29T06:40:16.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-167503724550 2-64424509457' 2024-03-29T06:40:16.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:16.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-03-29T06:40:16.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:16.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:40:16.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-03-29T06:40:16.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:16.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-03-29T06:40:16.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-03-29T06:40:16.863 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 206158430210 2024-03-29T06:40:16.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:17.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-03-29T06:40:17.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:40:18.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:40:18.344 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:18.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-03-29T06:40:18.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:18.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:18.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724550 2024-03-29T06:40:18.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:40:18.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724550 2024-03-29T06:40:18.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:18.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724550 2024-03-29T06:40:18.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2024-03-29T06:40:18.664 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 167503724550 2024-03-29T06:40:18.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:40:19.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724550 -lt 167503724550 2024-03-29T06:40:19.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:19.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-29T06:40:19.011 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:19.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:40:19.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-29T06:40:19.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:19.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-29T06:40:19.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-29T06:40:19.015 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509457 2024-03-29T06:40:19.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:40:19.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-29T06:40:19.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:40:19.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:19.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:19.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:40:19.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:40:19.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:40:19.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:40:19.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:40:19.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:40:19.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:40:19.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:40:20.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:40:20.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:40:20.059 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:20.060 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:20.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:40:20.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T06:40:20.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:40:20.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T06:40:20.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:40:20.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:40:20.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:40:20.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:40:20.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:40:20.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:40:20.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:40:20.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:40:20.538 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:40:20.538 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:40:20.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:40:20.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:40:20.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:20.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:21.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:40:21.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:40:21.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:40:21.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:40:21.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-29T06:40:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-29T06:40:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:40:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:40:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-29T06:40:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:40:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:40:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:40:21.609 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:40:21.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:40:22.119 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:40:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:40:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:40:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:40:22.606 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:40:22.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:40:22.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:40:22.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:40:22.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:40:22.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:40:22.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:40:22.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:40:23.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-29T06:40:26.016 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:40:26.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:40:26.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:40:26.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:40:26.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:40:26.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:40:26.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:40:26.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:40:26.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:40:26.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:40:26.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:40:26.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:40:26.554 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:40:26.554 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:40:26.554 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:40:26.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:40:26.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:40:26.556 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:40:26.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:40:26.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:40:26.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:40:26.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:40:26.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:40:26.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:40:26.589 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:26.586+0000 7fcca77cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:26.599 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:26.598+0000 7fcca77cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:26.601 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:26.598+0000 7fcca77cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:40:26.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:26.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:40:27.138 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:27.134+0000 7fcca77cf7c0 -1 Falling back to public interface 2024-03-29T06:40:27.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:28.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:28.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:28.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:40:28.204 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:40:28.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:28.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:40:28.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:29.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:29.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:29.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:40:29.513 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:40:29.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:29.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:40:29.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:30.049 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:30.046+0000 7fcca77cf7c0 -1 osd.1 52 log_to_monitors true 2024-03-29T06:40:30.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:30.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:30.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:40:30.838 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:40:30.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:30.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:40:31.264 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 57 up_thru 48 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/4165275841,v1:127.0.0.1:6811/4165275841] [v2:127.0.0.1:6812/4165275841,v1:127.0.0.1:6813/4165275841] exists,up 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:40:31.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:40:31.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:40:31.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:40:31.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:40:31.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:40:31.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:40:31.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:40:31.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:40:31.265 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:40:31.265 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:40:31.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:40:31.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:40:31.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:40:31.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:40:31.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:40:31.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:40:31.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:40:31.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:40:31.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:40:31.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:40:31.925 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:40:31.925 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:40:31.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:40:31.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:31.925 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:40:32.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430214 2024-03-29T06:40:32.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430214 2024-03-29T06:40:32.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214' 2024-03-29T06:40:32.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:32.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:40:32.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-03-29T06:40:32.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-03-29T06:40:32.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-244813135874' 2024-03-29T06:40:32.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:32.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:40:32.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-29T06:40:32.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-29T06:40:32.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430214 1-244813135874 2-64424509461' 2024-03-29T06:40:32.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:32.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430214 2024-03-29T06:40:32.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:32.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:40:32.957 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430214 2024-03-29T06:40:32.957 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:32.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430214 2024-03-29T06:40:32.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430214' 2024-03-29T06:40:32.959 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 206158430214 2024-03-29T06:40:32.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:33.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430213 -lt 206158430214 2024-03-29T06:40:33.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:40:34.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:40:34.359 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:34.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430214 2024-03-29T06:40:34.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:34.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-03-29T06:40:34.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:34.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:40:34.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-03-29T06:40:34.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:34.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-03-29T06:40:34.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-03-29T06:40:34.681 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135874 2024-03-29T06:40:34.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:40:35.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-03-29T06:40:35.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:35.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-29T06:40:35.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:35.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:40:35.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-29T06:40:35.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:35.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-29T06:40:35.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-29T06:40:35.009 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509461 2024-03-29T06:40:35.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:40:35.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-29T06:40:35.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:40:35.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:35.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:35.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:40:35.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:40:35.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:40:35.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:40:35.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:40:35.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:40:35.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:40:35.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:40:35.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:40:35.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:40:35.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:35.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:36.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:40:36.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:40:36.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:40:36.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:40:36.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:40:36.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:40:36.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:40:36.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:40:36.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:40:36.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:40:36.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:40:36.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:40:36.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:40:36.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:40:36.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:40:36.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:40:36.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:36.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-29T06:40:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:40:37.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:40:37.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:40:37.563 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:40:37.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:40:38.123 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:40:38.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:40:38.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:40:38.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:40:38.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-29T06:40:38.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-29T06:40:38.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:40:38.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-29T06:40:38.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-29T06:40:39.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-29T06:40:39.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:40:39.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:40:39.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:40:39.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:40:39.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:40:39.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:40:39.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:40:40.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:40:40.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:40:40.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:40:40.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:40:40.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:40:40.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:40:40.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:40:40.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:40:42.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:40:42.856 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:40:42.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:42.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:42.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:40:42.858 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:40:42.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:40:42.859 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:40:42.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:40:42.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:40:42.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:40:42.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:40:42.891 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:42.890+0000 7fe7c54907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:42.893 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:42.894+0000 7fe7c54907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:42.895 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:42.894+0000 7fe7c54907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:43.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:43.429 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:43.430+0000 7fe7c54907c0 -1 Falling back to public interface 2024-03-29T06:40:43.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:40:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:40:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:44.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:45.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:45.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:45.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:40:45.827 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:40:45.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:45.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:46.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:46.438 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:46.438+0000 7fe7c54907c0 -1 osd.0 62 log_to_monitors true 2024-03-29T06:40:47.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:47.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:47.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:40:47.149 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:40:47.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:47.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:40:47.741 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 67 up_thru 58 down_at 63 last_clean_interval [48,62) [v2:127.0.0.1:6802/932582749,v1:127.0.0.1:6803/932582749] [v2:127.0.0.1:6804/932582749,v1:127.0.0.1:6805/932582749] exists,up 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:40:47.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:40:47.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:40:47.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:40:47.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:40:47.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:40:47.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:40:47.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:40:47.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:40:47.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:40:47.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:40:47.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:40:47.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:40:47.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:40:47.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:40:47.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:40:47.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:40:47.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:40:47.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:40:47.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:40:48.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:40:48.299 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:40:48.300 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:40:48.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:40:48.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:48.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:40:48.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-29T06:40:48.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-29T06:40:48.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834' 2024-03-29T06:40:48.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:48.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:40:48.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-03-29T06:40:48.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-03-29T06:40:48.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-244813135878' 2024-03-29T06:40:48.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:40:48.909 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:40:49.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-29T06:40:49.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-29T06:40:49.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-244813135878 2-64424509465' 2024-03-29T06:40:49.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:49.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808834 2024-03-29T06:40:49.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:49.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:40:49.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808834 2024-03-29T06:40:49.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:49.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-29T06:40:49.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-03-29T06:40:49.057 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 287762808834 2024-03-29T06:40:49.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:49.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 287762808834 2024-03-29T06:40:49.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:40:50.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:40:50.407 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:40:50.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-29T06:40:50.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:50.757 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135878 2024-03-29T06:40:50.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:50.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:40:50.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135878 2024-03-29T06:40:50.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:50.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-03-29T06:40:50.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-03-29T06:40:50.763 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135878 2024-03-29T06:40:50.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:40:51.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135878 2024-03-29T06:40:51.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:40:51.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:40:51.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-29T06:40:51.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:40:51.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-29T06:40:51.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:40:51.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-29T06:40:51.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-29T06:40:51.201 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 64424509465 2024-03-29T06:40:51.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:40:51.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509466 -lt 64424509465 2024-03-29T06:40:51.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:40:51.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:51.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:51.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:40:51.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:40:51.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:40:51.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:40:51.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:40:51.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:40:51.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:40:51.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:40:52.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:40:52.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:40:52.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:40:52.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:40:52.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:40:52.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:40:52.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:40:52.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:40:52.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:40:52.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:40:52.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:40:53.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:40:53.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:40:53.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:40:53.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:40:53.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:40:53.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:40:53.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:40:53.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-29T06:40:56.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:40:56.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:40:56.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:40:56.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:40:56.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:40:56.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:40:56.509 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:40:56.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:40:56.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:40:56.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:40:56.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:40:56.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:40:56.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:40:56.512 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.2 2024-03-29T06:40:56.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:40:56.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:40:56.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:40:56.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:40:56.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:40:56.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:40:56.544 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:56.546+0000 7f28a8db67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:56.547 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:56.550+0000 7f28a8db67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:56.549 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:56.550+0000 7f28a8db67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:40:56.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:40:56.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:40:56.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:40:56.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:40:56.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:40:56.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:56.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:40:56.833 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:40:56.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:56.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:40:57.086 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:40:57.086+0000 7f28a8db67c0 -1 Falling back to public interface 2024-03-29T06:40:57.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:58.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:58.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:58.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:40:58.148 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:40:58.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:58.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:40:58.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:40:59.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:40:59.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:40:59.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:40:59.498 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:40:59.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:40:59.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:40:59.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:00.002 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:00.002+0000 7f28a8db67c0 -1 osd.2 69 log_to_monitors true 2024-03-29T06:41:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:41:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:41:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:00.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:41:01.146 INFO:tasks.workunit.client.0.smithi123.stderr:osd.2 up in weight 1 up_from 74 up_thru 67 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/2036986142,v1:127.0.0.1:6819/2036986142] [v2:127.0.0.1:6820/2036986142,v1:127.0.0.1:6821/2036986142] exists,up 47883a79-6a81-4db0-8169-4e454edea1fd 2024-03-29T06:41:01.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:41:01.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:41:01.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:41:01.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:41:01.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:41:01.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:41:01.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:41:01.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:41:01.148 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:41:01.148 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:41:01.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:41:01.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:41:01.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:41:01.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:41:01.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:41:01.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:41:01.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:41:01.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:41:01.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:41:01.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:41:01.726 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:01.726 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:41:01.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:41:01.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:01.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:41:01.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808838 2024-03-29T06:41:01.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808838 2024-03-29T06:41:01.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808838' 2024-03-29T06:41:01.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:01.874 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:41:02.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135882 2024-03-29T06:41:02.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135882 2024-03-29T06:41:02.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808838 1-244813135882' 2024-03-29T06:41:02.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:02.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:41:02.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-29T06:41:02.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-29T06:41:02.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808838 1-244813135882 2-317827579906' 2024-03-29T06:41:02.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:02.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808838 2024-03-29T06:41:02.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:02.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:41:02.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808838 2024-03-29T06:41:02.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:02.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808838 2024-03-29T06:41:02.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808838' 2024-03-29T06:41:02.268 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 287762808838 2024-03-29T06:41:02.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:41:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808836 -lt 287762808838 2024-03-29T06:41:02.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:41:03.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:41:03.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:41:04.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808838 -lt 287762808838 2024-03-29T06:41:04.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:04.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135882 2024-03-29T06:41:04.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:04.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:41:04.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135882 2024-03-29T06:41:04.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:04.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135882 2024-03-29T06:41:04.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135882' 2024-03-29T06:41:04.073 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135882 2024-03-29T06:41:04.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:41:04.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135882 -lt 244813135882 2024-03-29T06:41:04.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:04.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:04.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-29T06:41:04.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:41:04.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-29T06:41:04.425 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:04.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-29T06:41:04.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-29T06:41:04.426 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 317827579906 2024-03-29T06:41:04.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:41:04.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-29T06:41:04.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:41:04.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:04.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:05.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:41:05.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:05.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:05.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:05.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:05.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:05.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:05.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:05.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:41:05.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:05.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:05.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:05.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:41:05.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T06:41:05.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:41:05.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T06:41:05.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:41:05.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:41:05.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:41:05.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:05.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:05.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:05.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:05.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:05.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:05.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:06.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:41:06.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:06.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:06.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:06.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:41:06.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 2 2024-03-29T06:41:06.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:41:06.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:41:06.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:41:06.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:41:06.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:41:06.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:06.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:06.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:06.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:06.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:06.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:06.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:07.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:41:07.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:07.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:07.040 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:07.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:41:07.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-29T06:41:07.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:41:07.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:41:07.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:41:07.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:41:07.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:41:07.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:41:07.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:41:07.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:41:07.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:41:07.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:41:07.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:41:07.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:41:07.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:41:07.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:07.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:07.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:07.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:07.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:07.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:07.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:08.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:41:08.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:08.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:08.283 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-29T06:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:41:08.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:41:08.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:41:09.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-29T06:41:12.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-29T06:41:12.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:41:12.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:41:12.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:41:12.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:41:12.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:41:12.108 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:41:12.108 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:41:12.108 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:41:12.108 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:12.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:12.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:41:12.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:41:12.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:41:12.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:41:12.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:41:12.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:41:12.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:41:12.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:41:12.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:41:12.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:41:12.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:41:12.110 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.2 2024-03-29T06:41:12.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:41:12.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:41:12.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:41:12.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:41:12.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:41:12.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:41:12.142 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:12.141+0000 7f6216acb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:12.144 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:12.145+0000 7f6216acb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:12.146 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:12.145+0000 7f6216acb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:12.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:12.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:41:12.694 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:12.693+0000 7f6216acb7c0 -1 Falling back to public interface 2024-03-29T06:41:12.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:13.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:13.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:13.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:41:13.744 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:13.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:13.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:41:14.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:15.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:15.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:15.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:41:15.053 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:41:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:15.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:41:15.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:15.724 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:15.721+0000 7f6216acb7c0 -1 osd.2 76 log_to_monitors true 2024-03-29T06:41:16.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:16.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:16.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:41:16.365 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:41:16.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:16.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:41:16.828 INFO:tasks.workunit.client.0.smithi123.stderr:osd.2 up in weight 1 up_from 80 up_thru 74 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/3617132657,v1:127.0.0.1:6819/3617132657] [v2:127.0.0.1:6820/3617132657,v1:127.0.0.1:6821/3617132657] exists,up 47883a79-6a81-4db0-8169-4e454edea1fd 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:41:16.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:41:16.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:41:16.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:41:16.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:41:16.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:41:16.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:41:16.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:41:16.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:41:16.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:41:16.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:41:16.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:41:16.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:41:17.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:41:17.394 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:17.394 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:41:17.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:41:17.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:17.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:41:17.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808842 2024-03-29T06:41:17.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808842 2024-03-29T06:41:17.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808842' 2024-03-29T06:41:17.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:17.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:41:17.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135886 2024-03-29T06:41:17.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135886 2024-03-29T06:41:17.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808842 1-244813135886' 2024-03-29T06:41:17.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:17.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:41:17.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-03-29T06:41:17.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-03-29T06:41:17.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808842 1-244813135886 2-343597383682' 2024-03-29T06:41:17.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:17.981 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808842 2024-03-29T06:41:17.981 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:17.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:41:17.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808842 2024-03-29T06:41:17.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:17.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808842 2024-03-29T06:41:17.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808842' 2024-03-29T06:41:17.984 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 287762808842 2024-03-29T06:41:17.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:41:18.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808840 -lt 287762808842 2024-03-29T06:41:18.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:41:19.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:41:19.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:41:19.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808842 -lt 287762808842 2024-03-29T06:41:19.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:19.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135886 2024-03-29T06:41:19.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:19.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:41:19.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135886 2024-03-29T06:41:19.740 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:19.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135886 2024-03-29T06:41:19.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135886' 2024-03-29T06:41:19.741 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 244813135886 2024-03-29T06:41:19.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:41:20.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135886 -lt 244813135886 2024-03-29T06:41:20.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:20.078 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-03-29T06:41:20.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:20.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:41:20.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-03-29T06:41:20.081 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:20.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-03-29T06:41:20.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-03-29T06:41:20.083 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 343597383682 2024-03-29T06:41:20.083 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:41:20.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-03-29T06:41:20.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:41:20.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:20.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:20.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:41:20.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:20.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:20.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:20.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:20.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:20.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:20.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:21.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:41:21.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:21.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:21.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:21.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:41:21.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T06:41:21.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:41:21.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T06:41:21.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:41:21.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:41:21.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:41:21.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:21.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:21.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:21.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:21.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:21.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:21.562 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:21.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:41:21.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:21.853 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:21.853 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:22.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:41:22.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-29T06:41:22.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:41:22.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:41:22.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:41:22.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:41:22.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:41:22.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:41:22.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:41:22.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:41:22.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:41:22.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:41:22.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:41:22.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:41:22.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:41:22.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:22.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:22.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:22.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:22.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:22.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:22.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:23.152 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:41:23.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:23.154 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:23.154 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:23.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:41:23.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:41:23.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:41:23.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:41:23.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:41:23.995 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:41:24.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:41:24.572 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:41:24.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:41:24.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:41:24.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:41:24.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-29T06:41:24.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:41:24.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:41:24.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:2+0 records in 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:2+0 records out 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000163938 s, 25.0 MB/s 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:41:24.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:41:25.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:41:25.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:41:25.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:41:25.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:41:25.001 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:41:25.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:41:25.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:41:25.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:41:26.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:41:26.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:41:26.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:41:26.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:41:26.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:41:26.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:41:26.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:41:26.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:41:29.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:41:29.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:41:29.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:41:29.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:41:29.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:41:29.232 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:41:29.232 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:41:29.232 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:41:29.232 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:41:29.232 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:41:29.232 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:41:29.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:41:29.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:41:29.234 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:41:29.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:41:29.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:41:29.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:41:29.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:41:29.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:41:29.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:41:29.268 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:29.265+0000 7fec4fc8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:29.270 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:29.269+0000 7fec4fc8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:29.271 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:29.269+0000 7fec4fc8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:29.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:41:29.806 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:29.805+0000 7fec4fc8e7c0 -1 Falling back to public interface 2024-03-29T06:41:29.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:30.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:41:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:30.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:41:31.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:32.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:32.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:32.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:41:32.181 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:41:32.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:32.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:41:32.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:32.893 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:32.893+0000 7fec4fc8e7c0 -1 osd.1 84 log_to_monitors true 2024-03-29T06:41:33.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:41:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:41:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:41:34.071 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 89 up_thru 80 down_at 85 last_clean_interval [57,84) [v2:127.0.0.1:6810/2236990592,v1:127.0.0.1:6811/2236990592] [v2:127.0.0.1:6812/2236990592,v1:127.0.0.1:6813/2236990592] exists,up 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:41:34.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:41:34.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:41:34.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:41:34.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:41:34.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:41:34.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:41:34.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:41:34.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:41:34.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:41:34.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:41:34.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:41:34.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:41:34.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:41:34.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:41:34.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:41:34.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:41:34.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:41:34.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:41:34.176 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:41:34.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:41:34.726 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:34.726 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:41:34.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:41:34.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:34.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:41:35.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808846 2024-03-29T06:41:35.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808846 2024-03-29T06:41:35.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808846' 2024-03-29T06:41:35.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:35.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:41:35.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-03-29T06:41:35.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-03-29T06:41:35.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808846 1-382252089346' 2024-03-29T06:41:35.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:35.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:41:35.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383686 2024-03-29T06:41:35.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383686 2024-03-29T06:41:35.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808846 1-382252089346 2-343597383686' 2024-03-29T06:41:35.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:35.559 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808846 2024-03-29T06:41:35.559 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:35.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:41:35.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808846 2024-03-29T06:41:35.561 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:35.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808846 2024-03-29T06:41:35.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808846' 2024-03-29T06:41:35.563 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 287762808846 2024-03-29T06:41:35.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:41:35.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808845 -lt 287762808846 2024-03-29T06:41:35.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:41:36.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:41:36.934 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:41:37.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808846 -lt 287762808846 2024-03-29T06:41:37.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:37.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-03-29T06:41:37.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:37.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:41:37.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-03-29T06:41:37.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:37.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-03-29T06:41:37.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-03-29T06:41:37.246 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 382252089346 2024-03-29T06:41:37.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:41:37.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-03-29T06:41:37.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:37.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:37.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383686 2024-03-29T06:41:37.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:41:37.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383686 2024-03-29T06:41:37.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:37.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383686 2024-03-29T06:41:37.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383686' 2024-03-29T06:41:37.575 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 343597383686 2024-03-29T06:41:37.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:41:37.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383687 -lt 343597383686 2024-03-29T06:41:37.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:41:37.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:37.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:38.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:41:38.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:38.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:38.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:38.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:38.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:38.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:38.308 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:38.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:41:38.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:38.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:38.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:39.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:41:39.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:41:39.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:41:39.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:41:39.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-29T06:41:39.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-29T06:41:39.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:41:39.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:41:39.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-29T06:41:39.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:41:39.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:41:39.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:41:39.431 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:41:39.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:41:39.900 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:41:39.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:41:39.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:41:39.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:41:40.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:41:40.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:41:40.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:41:40.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:41:40.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:41:40.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:41:40.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:41:40.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:41:41.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:41:41.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:41:41.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:41:41.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:41:41.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:41:41.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:41:41.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:41:41.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-29T06:41:44.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:41:44.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:41:44.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:41:44.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:41:44.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:41:44.151 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:41:44.152 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:41:44.153 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:41:44.153 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:41:44.153 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:41:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:41:44.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:41:44.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:41:44.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:41:44.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:41:44.155 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:41:44.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:41:44.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:41:44.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:41:44.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:41:44.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:41:44.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:41:44.188 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:44.184+0000 7f98e0ad07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:44.190 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:44.188+0000 7f98e0ad07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:44.192 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:44.192+0000 7f98e0ad07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:44.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:41:44.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:41:44.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:41:44.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:41:44.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:41:44.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:44.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:41:44.509 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:41:44.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:41:44.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:44.726 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:44.724+0000 7f98e0ad07c0 -1 Falling back to public interface 2024-03-29T06:41:44.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:45.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:45.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:45.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:41:45.837 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:45.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:45.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:41:46.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:47.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:47.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:47.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:41:47.159 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:41:47.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:47.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:41:47.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:47.704 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:47.704+0000 7f98e0ad07c0 -1 osd.0 93 log_to_monitors true 2024-03-29T06:41:48.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:48.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:48.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:41:48.482 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:41:48.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:48.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:41:48.798 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 98 up_thru 89 down_at 94 last_clean_interval [67,93) [v2:127.0.0.1:6802/509562885,v1:127.0.0.1:6803/509562885] [v2:127.0.0.1:6804/509562885,v1:127.0.0.1:6805/509562885] exists,up 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:41:48.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:41:48.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:41:48.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:41:48.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:41:48.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:41:48.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:41:48.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:41:48.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:41:48.799 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:41:48.799 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:41:48.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:41:48.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:41:48.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:41:48.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:41:48.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:41:48.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:41:48.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:41:48.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:41:48.898 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:41:49.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:41:49.511 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:49.511 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:41:49.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:41:49.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:49.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:41:49.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-03-29T06:41:49.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-03-29T06:41:49.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010' 2024-03-29T06:41:49.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:49.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:41:50.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089350 2024-03-29T06:41:50.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089350 2024-03-29T06:41:50.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089350' 2024-03-29T06:41:50.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:41:50.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:41:50.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383690 2024-03-29T06:41:50.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383690 2024-03-29T06:41:50.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089350 2-343597383690' 2024-03-29T06:41:50.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:50.309 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795010 2024-03-29T06:41:50.309 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:50.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:41:50.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795010 2024-03-29T06:41:50.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:50.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-03-29T06:41:50.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-03-29T06:41:50.312 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 420906795010 2024-03-29T06:41:50.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:41:50.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-03-29T06:41:50.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:50.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089350 2024-03-29T06:41:50.798 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:50.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:41:50.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089350 2024-03-29T06:41:50.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:50.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089350 2024-03-29T06:41:50.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2024-03-29T06:41:50.802 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 382252089350 2024-03-29T06:41:50.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:41:51.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089350 -lt 382252089350 2024-03-29T06:41:51.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:41:51.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383690 2024-03-29T06:41:51.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:41:51.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:41:51.226 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:41:51.226 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383690 2024-03-29T06:41:51.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383690 2024-03-29T06:41:51.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383690' 2024-03-29T06:41:51.227 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 343597383690 2024-03-29T06:41:51.227 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:41:51.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383689 -lt 343597383690 2024-03-29T06:41:51.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:41:52.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:41:52.599 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:41:52.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383691 -lt 343597383690 2024-03-29T06:41:52.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:41:52.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:52.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:53.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:41:53.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:53.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:53.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:53.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:53.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:53.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:53.320 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:53.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:41:53.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:53.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:53.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:53.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:41:53.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T06:41:53.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:41:53.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T06:41:53.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:41:53.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:41:54.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:41:54.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:41:54.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:41:54.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:41:54.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:41:54.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:41:54.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:41:54.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:41:54.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:41:54.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:41:54.369 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:41:54.369 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:41:54.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:41:54.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:41:54.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:41:55.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:41:55.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:41:55.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:41:55.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:41:55.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:41:55.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:41:55.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:41:55.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-29T06:41:58.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:41:58.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:41:58.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:41:58.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:41:58.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:41:58.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:41:58.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:41:58.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:41:58.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:41:58.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:41:58.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:41:58.252 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:41:58.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:41:58.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:41:58.254 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:41:58.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:41:58.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:41:58.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:41:58.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:41:58.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:41:58.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:41:58.287 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:58.284+0000 7f8e6ecfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:58.304 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:58.304+0000 7f8e6ecfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:58.305 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:58.304+0000 7f8e6ecfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:41:58.569 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:41:58.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:58.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:41:58.842 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:41:58.840+0000 7f8e6ecfb7c0 -1 Falling back to public interface 2024-03-29T06:41:58.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:41:59.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:41:59.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:41:59.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:41:59.936 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:41:59.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:41:59.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:42:00.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:01.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:01.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:01.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:42:01.319 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:42:01.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:01.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:42:01.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:01.661 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:01.660+0000 7f8e6ecfb7c0 -1 osd.1 100 log_to_monitors true 2024-03-29T06:42:02.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:02.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:02.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:42:02.638 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:42:02.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:02.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:42:03.144 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 105 up_thru 105 down_at 101 last_clean_interval [89,100) [v2:127.0.0.1:6810/2406499328,v1:127.0.0.1:6811/2406499328] [v2:127.0.0.1:6812/2406499328,v1:127.0.0.1:6813/2406499328] exists,up 50ac9ad8-fdf4-4271-948a-7b35bd3f9150 2024-03-29T06:42:03.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:42:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:42:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:42:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:42:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:42:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:42:03.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:42:03.145 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:42:03.146 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:42:03.146 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:42:03.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:42:03.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:42:03.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:42:03.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:42:03.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:42:03.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:42:03.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:42:03.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:42:03.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:42:03.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:42:03.796 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:42:03.796 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:42:03.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:42:03.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:03.797 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:42:03.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795014 2024-03-29T06:42:03.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795014 2024-03-29T06:42:03.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795014' 2024-03-29T06:42:03.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:03.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:42:04.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-29T06:42:04.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-29T06:42:04.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795014 1-450971566082' 2024-03-29T06:42:04.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:04.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:42:04.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383694 2024-03-29T06:42:04.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383694 2024-03-29T06:42:04.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795014 1-450971566082 2-343597383694' 2024-03-29T06:42:04.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:04.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795014 2024-03-29T06:42:04.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:04.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:42:04.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:04.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795014 2024-03-29T06:42:04.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795014 2024-03-29T06:42:04.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795014' 2024-03-29T06:42:04.364 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 420906795014 2024-03-29T06:42:04.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:42:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795014 -lt 420906795014 2024-03-29T06:42:04.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:04.714 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-450971566082 2024-03-29T06:42:04.714 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:04.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:42:04.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-450971566082 2024-03-29T06:42:04.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-29T06:42:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 450971566082' 2024-03-29T06:42:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 450971566082 2024-03-29T06:42:04.719 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:42:05.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-03-29T06:42:05.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:05.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383694 2024-03-29T06:42:05.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:05.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:42:05.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383694 2024-03-29T06:42:05.032 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:05.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383694 2024-03-29T06:42:05.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383694' 2024-03-29T06:42:05.033 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 343597383694 2024-03-29T06:42:05.034 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:42:05.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383693 -lt 343597383694 2024-03-29T06:42:05.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:42:06.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:42:06.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:42:06.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383695 -lt 343597383694 2024-03-29T06:42:06.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:42:06.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:42:06.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:42:07.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:42:07.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:42:07.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:42:07.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:42:07.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:42:07.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:42:07.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:42:07.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:42:07.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:42:07.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:42:07.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:42:07.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:42:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:42:07.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-29T06:42:07.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:42:07.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:42:07.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:42:08.232 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:42:08.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:42:08.835 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:42:08.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:42:08.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:42:08.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:42:09.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-29T06:42:09.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-29T06:42:09.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-29T06:42:09.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-29T06:42:09.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-29T06:42:09.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:42:09.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:42:09.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:42:09.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:42:09.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:42:09.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:42:09.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:42:09.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:42:09.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:42:09.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:42:09.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:42:12.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:42:12.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:42:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:42:12.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:42:12.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:42:12.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:42:12.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:42:12.684 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:42:12.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:42:12.685 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:42:12.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:42:12.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:42:12.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:42:12.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:42:12.719 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:12.716+0000 7f773c9707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:12.740 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:12.740+0000 7f773c9707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:12.741 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:12.740+0000 7f773c9707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:13.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:13.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:13.306 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:13.304+0000 7f773c9707c0 -1 Falling back to public interface 2024-03-29T06:42:13.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:14.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:14.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:14.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:42:14.391 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:42:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:14.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:14.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:15.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:15.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:15.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:42:15.709 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:42:15.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:15.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:16.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:16.233 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:16.232+0000 7f773c9707c0 -1 osd.0 109 log_to_monitors true 2024-03-29T06:42:17.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:42:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:42:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:17.386 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 114 up_thru 105 down_at 110 last_clean_interval [98,109) [v2:127.0.0.1:6802/2033993505,v1:127.0.0.1:6803/2033993505] [v2:127.0.0.1:6804/2033993505,v1:127.0.0.1:6805/2033993505] exists,up 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:42:17.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:42:17.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:42:17.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:42:17.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:42:17.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:42:17.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:42:17.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:42:17.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:42:17.387 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:42:17.388 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:42:17.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:42:17.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:42:17.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:42:17.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:42:17.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:42:17.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:42:17.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:42:17.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:42:17.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:42:17.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:42:17.917 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:42:17.917 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:42:17.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:42:17.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:17.918 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:42:18.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-03-29T06:42:18.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-03-29T06:42:18.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-489626271746' 2024-03-29T06:42:18.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:18.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:42:18.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566086 2024-03-29T06:42:18.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566086 2024-03-29T06:42:18.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-489626271746 1-450971566086' 2024-03-29T06:42:18.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:18.606 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:42:18.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383698 2024-03-29T06:42:18.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383698 2024-03-29T06:42:18.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-489626271746 1-450971566086 2-343597383698' 2024-03-29T06:42:18.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:18.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-489626271746 2024-03-29T06:42:18.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:18.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:42:18.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-489626271746 2024-03-29T06:42:18.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:18.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-03-29T06:42:18.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 489626271746' 2024-03-29T06:42:18.943 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 489626271746 2024-03-29T06:42:18.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:42:19.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 489626271746 2024-03-29T06:42:19.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:42:20.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:42:20.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:42:20.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271746 -lt 489626271746 2024-03-29T06:42:20.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:20.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-450971566086 2024-03-29T06:42:20.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:20.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:42:20.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-450971566086 2024-03-29T06:42:20.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:20.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566086 2024-03-29T06:42:20.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 450971566086' 2024-03-29T06:42:20.826 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 450971566086 2024-03-29T06:42:20.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:42:21.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566086 -lt 450971566086 2024-03-29T06:42:21.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:21.172 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383698 2024-03-29T06:42:21.172 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:21.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:42:21.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383698 2024-03-29T06:42:21.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:21.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383698 2024-03-29T06:42:21.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383698' 2024-03-29T06:42:21.177 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 343597383698 2024-03-29T06:42:21.177 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:42:21.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383698 -lt 343597383698 2024-03-29T06:42:21.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:42:21.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:42:21.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:42:21.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:42:21.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:42:21.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:42:21.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:42:21.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:42:21.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:42:21.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:42:21.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:42:22.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:42:22.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:42:22.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:42:22.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:42:22.613 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:42:22.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:42:22.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:42:22.615 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:42:22.615 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:42:22.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:42:22.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:42:22.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:42:22.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:42:22.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:42:22.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:42:22.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:42:22.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:42:22.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:42:22.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:42:22.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-29T06:42:26.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:42:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:42:26.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:42:26.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:42:26.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:42:26.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:42:26.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:42:26.457 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:42:26.457 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:26.457 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:26.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:26.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:42:26.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:42:26.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:42:26.459 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:42:26.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:42:26.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:42:26.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:42:26.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:42:26.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:42:26.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:42:26.493 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:26.491+0000 7fe0ca6cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:26.496 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:26.495+0000 7fe0ca6cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:26.497 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:26.495+0000 7fe0ca6cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:26.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:27.054 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:27.051+0000 7fe0ca6cf7c0 -1 Falling back to public interface 2024-03-29T06:42:27.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:28.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:28.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:28.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:42:28.101 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:42:28.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:28.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:28.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:29.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:29.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:29.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:42:29.424 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:42:29.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:29.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:29.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:42:30.055 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:30.051+0000 7fe0ca6cf7c0 -1 osd.0 117 log_to_monitors true 2024-03-29T06:42:30.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:42:30.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:42:30.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:42:30.745 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:42:30.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:42:30.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:42:31.333 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 122 up_thru 114 down_at 118 last_clean_interval [114,117) [v2:127.0.0.1:6802/2961161995,v1:127.0.0.1:6803/2961161995] [v2:127.0.0.1:6804/2961161995,v1:127.0.0.1:6805/2961161995] exists,up 4068ae4d-160b-4d7d-b085-8dd763ecada0 2024-03-29T06:42:31.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:42:31.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:42:31.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:42:31.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:42:31.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:42:31.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:42:31.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:42:31.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:42:31.335 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:42:31.335 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:42:31.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:42:31.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:42:31.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:42:31.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:42:31.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:42:31.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:42:31.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:42:31.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:42:31.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:42:31.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:42:31.938 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:42:31.938 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:42:31.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:42:31.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:31.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:42:32.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-03-29T06:42:32.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-03-29T06:42:32.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-03-29T06:42:32.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:32.326 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:42:32.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566090 2024-03-29T06:42:32.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566090 2024-03-29T06:42:32.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-450971566090' 2024-03-29T06:42:32.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:42:32.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:42:32.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383702 2024-03-29T06:42:32.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383702 2024-03-29T06:42:32.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-450971566090 2-343597383702' 2024-03-29T06:42:32.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:32.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-03-29T06:42:32.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:32.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:42:32.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-03-29T06:42:32.897 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:32.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-03-29T06:42:32.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-03-29T06:42:32.899 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 523986010114 2024-03-29T06:42:32.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:42:33.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-03-29T06:42:33.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:42:34.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:42:34.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:42:34.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-03-29T06:42:34.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:34.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-450971566090 2024-03-29T06:42:34.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:34.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:42:34.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-450971566090 2024-03-29T06:42:34.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:34.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566090 2024-03-29T06:42:34.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 450971566090' 2024-03-29T06:42:34.548 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 450971566090 2024-03-29T06:42:34.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:42:34.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566090 -lt 450971566090 2024-03-29T06:42:34.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:42:34.896 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:42:34.896 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383702 2024-03-29T06:42:34.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:42:34.898 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383702 2024-03-29T06:42:34.898 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:42:34.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383702 2024-03-29T06:42:34.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383702' 2024-03-29T06:42:34.899 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.2 seq 343597383702 2024-03-29T06:42:34.900 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:42:35.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383702 -lt 343597383702 2024-03-29T06:42:35.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:42:35.212 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:42:35.212 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:42:35.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:42:35.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:42:35.578 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:42:35.578 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:42:35.578 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:42:35.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:42:35.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:42:35.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:42:35.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:42:35.902 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:42:35.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:42:35.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:42:36.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:42:36.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-29T06:42:36.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:42:36.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:42:36.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:42:36.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:42:36.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:42:36.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:42:36.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:42:36.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:42:36.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:42:36.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:42:36.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:42:36.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:42:36.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:42:36.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:42:36.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:42:36.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:42:37.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:42:37.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:42:37.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:42:37.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-29T06:42:37.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-29T06:42:37.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:42:37.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-29T06:42:37.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-29T06:42:37.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:42:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-29T06:42:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-29T06:42:37.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-29T06:42:37.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-29T06:42:37.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:37.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:42:37.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:37.409 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:42:37.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:37.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-29T06:42:38.029 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-29T06:42:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:42:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:38.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:42:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:42:38.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:38.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:42:38.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:38.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:42:38.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:38.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:42:38.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:39:37.023697+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:38.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:42:39.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:42:39.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:39.344 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:42:39.344 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:39.344 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:42:39.345 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:39.345 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:42:39.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:39:37.023697+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:39.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:42:40.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:42:40.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:40.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:42:40.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:40.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:42:40.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:40.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:42:40.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:39:37.023697+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:40.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:42:41.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:42:41.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:41.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:42:41.988 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:41.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:42:41.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:41.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:42:42.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:42:38.648855+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:42.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:42:42.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-29T06:42:42.343 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-29T06:42:42.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-29T06:42:42.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-29T06:42:42.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-29T06:42:42.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-29T06:42:42.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-29T06:42:42.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-03-29T06:42:42.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:42:42.508 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:42:42.508 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:42:42.508 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:42:42.508 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:42:42.508 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:42:42.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:42:42.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:42:42.551 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:42:42.551 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:42:42.551 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:42:42.551 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:42:42.551 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:42:42.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:42:42.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:42:42.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:42:42.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-29T06:42:42.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-29T06:42:42.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-29T06:42:42.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-29T06:42:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:42:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:42:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:42.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:42:42.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:42.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-29T06:42:43.232 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-29T06:42:43.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-29T06:39:37.023697+0000 last_deep_scrub_stamp 2024-03-29T06:42:43.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:42:43.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:43.247 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-29T06:42:43.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:42:43.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:43.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:42:43.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:43.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:42:43.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:43.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:42:43.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:39:37.023697+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:43.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:42:44.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:42:44.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:44.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:42:44.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:44.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:42:44.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:44.531 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:42:44.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:39:37.023697+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:44.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:42:45.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:42:45.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:45.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:42:45.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:45.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:42:45.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:45.850 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:42:46.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:39:37.023697+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:46.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:42:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:42:47.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:47.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:42:47.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:47.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:42:47.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:42:47.169 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:47.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:39:37.023697+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:47.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:42:48.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:42:48.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:42:48.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:42:48.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:42:48.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:42:48.478 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:42:48.479 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:42:48.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:42:44.049186+0000 '>' 2024-03-29T06:39:37.023697+0000 2024-03-29T06:42:48.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:42:48.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-29T06:42:48.814 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-29T06:42:48.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-29T06:42:48.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-29T06:42:48.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-29T06:42:48.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-29T06:42:48.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-29T06:42:48.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-03-29T06:42:48.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-03-29T06:42:48.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:42:48.923 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:42:48.923 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:42:48.923 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:42:48.923 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:42:48.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:42:48.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-29T06:42:48.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-29T06:42:48.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:42:49.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:42:49.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:42:49.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-29T06:42:49.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-29T06:42:49.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-29T06:42:49.405 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' removed 2024-03-29T06:42:49.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:42:49.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:42:49.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:42:49.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:42:49.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:42:49.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:42:49.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:42:49.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:42:49.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:42:49.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:42:49.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:42:49.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:42:49.743 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:42:49.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:42:49.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:42:49.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:42:49.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:42:49.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:42:49.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:42:49.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:42:49.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:42:49.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:42:49.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:42:49.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:42:49.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:49.792 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:49.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:42:49.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:42:49.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:42:49.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:42:49.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:42:49.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:42:49.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:42:49.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:42:49.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:42:49.795 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:42:49.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:42:49.796 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:42:49.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:42:49.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:42:49.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:42:49.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:42:49.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:42:49.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:42:49.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:42:49.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:42:49.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:42:49.802 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:42:49.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:42:49.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:42:49.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:42:49.805 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:42:49.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:42:49.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:42:49.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:42:49.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:42:49.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:49.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:49.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:42:49.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:42:49.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:42:49.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:42:49.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:42:49.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:49.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:49.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:42:49.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:42:49.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:42:49.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:42:49.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:42:49.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:42:49.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-29T06:42:50.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:42:50.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:42:50.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:42:50.299 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:42:50.299 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:50.299 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:50.300 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:50.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-29T06:42:50.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:42:50.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:42:50.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:42:50.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:42:50.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:42:50.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:42:50.545 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:42:50.545 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:42:50.545 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:42:50.546 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:42:50.546 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:50.546 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:50.546 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:42:50.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:42:50.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:42:50.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:42:50.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:42:50.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:42:50.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:42:50.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:42:50.664 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:42:50.664 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:42:50.664 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:42:50.665 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:42:50.665 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:50.665 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:50.665 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:42:50.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:42:50.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:42:50.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-29T06:42:50.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:42:50.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:42:50.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:42:50.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:42:50.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:42:50.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:42:51.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:42:51.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:42:51.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:42:51.668 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:42:51.668 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:51.668 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:51.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:51.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:42:51.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:42:51.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-03-29T06:42:51.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:42:51.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:42:51.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:42:51.708 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:42:51.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:42:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:42:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:42:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:42:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:42:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:42:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:42:51.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:42:51.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:42:51.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:42:51.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 72c5bf1f-765a-483b-97a4-bdfd228a01f9' 2024-03-29T06:42:51.711 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:42:51.712 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:42:51.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrYgZm1t5vKxAAgZeqrhv1rDyoghhr8EyohA== 2024-03-29T06:42:51.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrYgZm1t5vKxAAgZeqrhv1rDyoghhr8EyohA=="}' 2024-03-29T06:42:51.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72c5bf1f-765a-483b-97a4-bdfd228a01f9 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:42:52.053 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:42:52.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:42:52.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrYgZm1t5vKxAAgZeqrhv1rDyoghhr8EyohA== --osd-uuid 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:42:52.108 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:52.107+0000 7f7a0aa6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:52.155 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:52.151+0000 7f7a0aa6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:52.156 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:52.155+0000 7f7a0aa6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:42:52.157 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:52.155+0000 7f7a0aa6a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:42:52.157 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:52.155+0000 7f7a0aa6a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:42:52.157 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:52.155+0000 7f7a0aa6a7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:42:52.263 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:42:52.259+0000 7f7a0aa6a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:43:00.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:43:00.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:43:00.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:43:00.546 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:43:00.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:43:00.938 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:43:00.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:43:00.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:43:00.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:43:00.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:43:00.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:43:00.970 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:00.966+0000 7f849df847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:01.026 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:01.022+0000 7f849df847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:01.027 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:01.022+0000 7f849df847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:01.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:43:01.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:43:01.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:43:01.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:43:01.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:43:01.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:01.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:43:01.250 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:43:01.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:01.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:01.560 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:01.617 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:01.614+0000 7f849df847c0 -1 Falling back to public interface 2024-03-29T06:43:02.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:02.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:02.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:43:02.563 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:43:02.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:02.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:02.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:03.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:03.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:03.872 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:43:03.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:43:03.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:03.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:04.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:04.530 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:04.526+0000 7f849df847c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:43:05.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:05.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:05.191 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:43:05.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:43:05.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:05.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:05.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:06.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:06.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:06.654 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:43:06.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:43:06.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:06.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:07.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:08.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:08.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:08.307 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:43:08.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:43:08.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:08.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:08.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:09.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:09.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:09.607 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:43:09.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:43:09.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:09.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:09.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:10.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:10.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:10.918 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:43:10.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:43:10.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:10.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:11.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:12.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:12.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:12.357 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:43:12.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:43:12.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:12.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:13.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:13.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:13.833 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:43:13.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:43:13.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:13.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:14.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:15.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:15.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:15.149 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:43:15.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:43:15.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:15.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:15.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:16.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:16.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:16.621 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:43:16.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:43:16.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:16.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:16.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:17.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:17.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:17.938 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:43:17.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:43:17.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:17.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:18.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:19.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:19.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:19.324 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:43:19.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:43:19.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:19.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:19.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:20.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:20.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:20.849 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:43:20.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:43:20.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:20.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:21.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:22.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:22.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:22.358 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:43:22.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:43:22.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:22.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:22.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:23.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:23.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:23.943 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:43:23.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:43:23.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:23.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:24.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:25.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:25.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:25.410 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:43:25.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:43:25.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:25.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:26.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:27.013 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:43:27.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:43:27.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:27.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:27.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:28.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:28.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:28.633 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:43:28.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:43:28.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:28.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:29.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:30.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:30.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:30.339 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:43:30.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:43:30.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:30.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:31.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:32.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:32.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:32.431 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:43:32.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:43:32.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:32.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:32.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:33.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:33.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:33.966 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:43:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:43:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:33.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:34.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:35.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:35.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:35.280 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:43:35.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:43:35.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:35.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:35.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:36.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:36.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:36.844 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:43:36.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:43:36.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:36.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:37.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:38.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:38.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:38.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:43:38.159 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:43:38.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:38.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:43:38.518 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1914929736,v1:127.0.0.1:6803/1914929736] [v2:127.0.0.1:6804/1914929736,v1:127.0.0.1:6805/1914929736] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:43:38.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:43:38.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:43:38.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:43:38.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:43:38.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:43:38.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:43:38.525 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:43:38.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:43:38.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4d3d9a88-c87a-446c-af5e-3e0717b128cf' 2024-03-29T06:43:38.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:43:38.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaYwZmufyoIBAAyhMslM7dBEMitfQqUViNEA== 2024-03-29T06:43:38.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaYwZmufyoIBAAyhMslM7dBEMitfQqUViNEA=="}' 2024-03-29T06:43:38.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d3d9a88-c87a-446c-af5e-3e0717b128cf -i td/osd-scrub-repair/1/new.json 2024-03-29T06:43:39.103 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:43:39.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:43:39.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaYwZmufyoIBAAyhMslM7dBEMitfQqUViNEA== --osd-uuid 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:43:39.158 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:39.153+0000 7f56678e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:39.246 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:39.245+0000 7f56678e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:39.248 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:39.249+0000 7f56678e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:39.249 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:39.249+0000 7f56678e77c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:43:39.249 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:39.249+0000 7f56678e77c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:43:39.249 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:39.249+0000 7f56678e77c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:43:39.333 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:39.333+0000 7f56678e77c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:43:47.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:43:47.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:43:47.619 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:43:47.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:43:47.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:43:48.015 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:43:48.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:43:48.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:43:48.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:43:48.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:43:48.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:43:48.041 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:48.041+0000 7f52ad32c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:48.043 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:48.041+0000 7f52ad32c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:48.045 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:48.045+0000 7f52ad32c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:43:48.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:43:48.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:43:48.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:43:48.328 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:43:48.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:43:48.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:43:48.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:48.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:43:48.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:48.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:48.583 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:48.581+0000 7f52ad32c7c0 -1 Falling back to public interface 2024-03-29T06:43:48.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:49.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:49.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:49.625 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:43:49.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:43:49.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:49.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:49.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:50.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:50.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:50.935 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:43:50.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:43:50.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:50.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:51.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:51.692 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:43:51.693+0000 7f52ad32c7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:43:52.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:52.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:52.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:43:52.242 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:43:52.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:52.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:52.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:53.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:53.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:53.815 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:43:53.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:43:53.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:53.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:54.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:55.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:55.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:55.233 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:43:55.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:43:55.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:55.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:55.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:56.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:56.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:56.651 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:43:56.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:43:56.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:56.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:57.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:58.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:58.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:58.107 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:43:58.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:43:58.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:58.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:58.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:43:59.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:43:59.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:43:59.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:43:59.408 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:43:59.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:43:59.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:43:59.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:00.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:00.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:00.884 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:44:00.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:44:00.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:00.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:01.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:02.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:02.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:02.336 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:44:02.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:44:02.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:02.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:02.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:03.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:03.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:03.655 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:44:03.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:44:03.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:03.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:04.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:05.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:05.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:05.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:44:05.100 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:44:05.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:05.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:05.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:06.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:06.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:06.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:44:06.552 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:44:06.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:06.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:06.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:07.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:07.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:07.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:44:07.927 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:44:07.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:07.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:08.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:09.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:09.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:09.237 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:44:09.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:44:09.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:09.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:09.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:10.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:10.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:10.623 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:44:10.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:44:10.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:10.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:10.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:11.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:11.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:11.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:44:11.983 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:44:11.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:11.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:12.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:13.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:13.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:13.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:44:13.375 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:44:13.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:13.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:14.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:15.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:15.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:15.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:44:15.213 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:44:15.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:15.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:15.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:16.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:16.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:16.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:44:16.712 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:44:16.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:16.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:17.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:18.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:18.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:18.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:44:18.995 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:44:18.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:18.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:19.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:20.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:20.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:20.846 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:44:20.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:44:20.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:20.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:21.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:22.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:22.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:22.496 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:44:22.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:44:22.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:22.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:23.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:24.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:24.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:24.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:44:24.120 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:44:24.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:24.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:24.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:44:25.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:44:25.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:44:25.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:44:25.916 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:44:25.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:44:25.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:44:26.234 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2697071079,v1:127.0.0.1:6811/2697071079] [v2:127.0.0.1:6812/2697071079,v1:127.0.0.1:6813/2697071079] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:44:26.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:44:26.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:44:26.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:44:26.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-03-29T06:44:26.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:44:26.682 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:44:26.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:44:26.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:44:27.231 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:44:27.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:44:28.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:44:31.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-29T06:44:31.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:44:31.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:44:31.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:44:31.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:44:31.245 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:44:31.245 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:44:31.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:44:31.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:44:31.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:44:31.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:44:31.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:44:31.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:44:31.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:44:31.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:44:31.379 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:44:31.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:44:31.689 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:44:31.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:44:31.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:44:31.690 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:44:31.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-29T06:44:31.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-29T06:44:31.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-29T06:44:31.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:44:31.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:44:31.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-29T06:44:31.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-29T06:44:31.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672967' 2024-03-29T06:44:31.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:44:31.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-29T06:44:31.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:44:31.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:44:31.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-29T06:44:31.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:44:31.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-29T06:44:31.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-29T06:44:31.979 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836499 2024-03-29T06:44:31.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:44:32.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-29T06:44:32.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:44:33.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:44:33.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:44:33.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-29T06:44:33.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:44:33.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-29T06:44:33.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:44:33.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:44:33.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-29T06:44:33.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:44:33.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-29T06:44:33.579 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672967 2024-03-29T06:44:33.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-29T06:44:33.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:44:33.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-03-29T06:44:33.894 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:44:33.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:44:33.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:44:34.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:44:34.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:44:34.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:44:34.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:44:34.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:44:34.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:44:34.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:44:34.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:44:34.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:44:34.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:44:34.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:44:34.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:44:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:44:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:44:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:44:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-03-29T06:44:34.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-29T06:44:35.348 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'foo' created 2024-03-29T06:44:35.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:44:36.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-03-29T06:44:36.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-29T06:44:37.263 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'csr_pool' created 2024-03-29T06:44:37.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:44:38.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-29T06:44:38.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:44:38.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:44:38.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:44:38.288 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:44:38.289 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:44:38.289 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:44:38.289 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:44:38.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:44:38.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:44:38.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:44:38.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:44:38.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:44:38.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:44:38.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:44:38.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:44:38.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:44:38.984 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:44:38.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:44:38.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:44:38.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:44:39.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-29T06:44:39.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-29T06:44:39.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-29T06:44:39.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:44:39.181 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:44:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-29T06:44:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-29T06:44:39.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672969' 2024-03-29T06:44:39.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:44:39.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-29T06:44:39.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:44:39.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:44:39.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-29T06:44:39.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:44:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-29T06:44:39.383 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836502 2024-03-29T06:44:39.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-29T06:44:39.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:44:39.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-29T06:44:39.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:44:40.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:44:40.673 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:44:41.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-29T06:44:41.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:44:41.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-29T06:44:41.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:44:41.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:44:41.034 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-29T06:44:41.034 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:44:41.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-29T06:44:41.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-29T06:44:41.036 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672969 2024-03-29T06:44:41.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:44:41.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-03-29T06:44:41.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:44:41.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:44:41.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:44:41.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:44:41.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:44:41.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:44:41.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:44:41.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:44:41.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:44:41.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:44:41.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:44:42.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:44:42.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:44:42.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:44:42.025 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:44:42.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:44:42.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:44:42.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:44:42.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:42.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:42.874 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:42.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:43.468 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:43.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:43.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:43.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:43.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-29T06:44:43.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-29T06:44:44.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:44.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-29T06:44:44.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-29T06:44:44.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:44.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:44.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-29T06:44:44.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:44.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:44.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:44.671 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:44.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:45.214 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:45.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:45.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:45.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:45.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-29T06:44:45.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:46.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:46.455 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:46.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:46.977 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:47.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:47.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:47.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:47.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-03-29T06:44:47.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-03-29T06:44:47.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:47.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:48.205 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:48.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:48.740 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:48.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:48.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:48.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:49.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-03-29T06:44:49.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-03-29T06:44:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-29T06:44:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-03-29T06:44:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:49.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-03-29T06:44:49.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:49.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:49.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:49.952 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:49.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:50.562 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:50.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:50.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:50.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:50.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-03-29T06:44:51.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-03-29T06:44:51.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:51.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-29T06:44:51.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-03-29T06:44:51.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:51.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:51.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-03-29T06:44:51.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:51.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:51.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:51.865 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:51.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:52.341 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:52.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:52.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:52.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:52.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-03-29T06:44:52.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:53.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:53.536 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:53.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:54.071 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:54.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:54.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:54.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:54.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-03-29T06:44:54.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-03-29T06:44:55.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:55.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:55.474 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:55.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:55.992 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:56.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:56.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:56.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:56.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-03-29T06:44:56.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:56.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:56.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:57.305 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:57.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:57.806 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:57.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:57.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:57.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:44:58.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-03-29T06:44:58.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-03-29T06:44:58.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:44:58.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-29T06:44:58.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-03-29T06:44:58.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:44:58.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:44:58.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-03-29T06:44:58.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:44:58.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:44:58.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:44:59.177 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:44:59.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:44:59.661 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:44:59.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:44:59.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:44:59.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:00.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-03-29T06:45:00.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-03-29T06:45:00.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:00.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:01.073 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:01.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:01.574 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:01.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:01.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:01.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:01.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-03-29T06:45:02.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:02.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:02.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:02.778 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:02.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:03.329 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:03.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:03.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:03.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:03.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-03-29T06:45:03.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-03-29T06:45:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-29T06:45:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-03-29T06:45:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:04.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-03-29T06:45:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:04.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:04.725 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:04.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:05.201 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:05.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:05.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:05.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:05.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-03-29T06:45:05.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:05.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:05.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:06.487 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:06.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:07.114 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:07.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:07.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:07.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:07.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-03-29T06:45:07.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:07.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:08.300 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:08.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:08.877 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:08.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:08.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:08.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:09.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-03-29T06:45:09.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:10.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:10.464 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:10.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:10.974 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:10.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:10.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:10.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:11.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-03-29T06:45:11.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:11.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:12.135 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:12.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:12.678 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:12.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:12.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:12.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-03-29T06:45:13.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:13.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:13.939 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:13.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:14.574 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:14.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:14.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:14.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:15.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-03-29T06:45:15.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-03-29T06:45:15.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:15.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-29T06:45:15.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-03-29T06:45:15.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:45:15.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:45:15.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-03-29T06:45:15.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:45:15.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:45:15.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:45:15.827 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:45:15.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:45:16.496 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:45:16.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:45:16.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:45:16.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:45:17.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-03-29T06:45:17.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-03-29T06:45:17.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-03-29T06:45:17.307 INFO:tasks.workunit.client.0.smithi123.stderr:1025+0 records in 2024-03-29T06:45:17.308 INFO:tasks.workunit.client.0.smithi123.stderr:1025+0 records out 2024-03-29T06:45:17.308 INFO:tasks.workunit.client.0.smithi123.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00328395 s, 320 MB/s 2024-03-29T06:45:17.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-03-29T06:45:17.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-03-29T06:45:17.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-29T06:45:17.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-29T06:45:17.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-29T06:45:17.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-29T06:45:17.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:45:17.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-29T06:45:17.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-03-29T06:45:17.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-29T06:45:17.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-29T06:45:17.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:45:17.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-29T06:45:18.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-03-29T06:45:18.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-03-29T06:45:18.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:45:18.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:45:18.161 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:45:18.161 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:45:18.161 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:45:18.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:45:18.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-29T06:45:18.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61544/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-29T06:45:18.265 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T06:45:18.265 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-29T06:45:18.265 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T06:45:18.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-03-29T06:45:18.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:45:18.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:45:18.279 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:45:18.279 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:45:18.279 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:45:18.279 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:45:18.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-29T06:45:18.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.61544/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-29T06:45:18.385 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T06:45:18.386 INFO:tasks.workunit.client.0.smithi123.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-29T06:45:18.386 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T06:45:18.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-29T06:45:18.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-29T06:45:18.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:45:18.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:45:18.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:45:18.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:45:18.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:45:18.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-29T06:44:36.841625+0000 2024-03-29T06:45:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-29T06:45:18.993 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-29T06:44:36.841625+0000 last_deep_scrub_stamp 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:44:36.841625+0000 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:45:19.009 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:45:19.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:45:19.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:45:19.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:44:36.841625+0000 '>' 2024-03-29T06:44:36.841625+0000 2024-03-29T06:45:19.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:45:20.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:45:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:45:20.324 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:45:20.325 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:45:20.325 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:45:20.325 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:45:20.325 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:45:20.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:44:36.841625+0000 '>' 2024-03-29T06:44:36.841625+0000 2024-03-29T06:45:20.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:45:21.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:45:21.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:45:21.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:45:21.630 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:45:21.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:45:21.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:45:21.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:45:21.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:44:36.841625+0000 '>' 2024-03-29T06:44:36.841625+0000 2024-03-29T06:45:21.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:45:22.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:45:22.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:45:22.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:45:22.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:45:22.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:45:22.949 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:45:22.949 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:45:23.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:44:36.841625+0000 2024-03-29T06:45:23.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:45:23.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-29T06:45:23.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:23.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-29T06:45:23.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-03-29T06:45:23.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:45:23.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:45:23.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:45:23.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:45:23.269 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:45:23.269 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:45:23.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:45:23.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:45:23.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:45:23.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:45:23.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:23.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:45:23.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:45:23.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:45:23.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:45:23.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:45:23.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:25.733 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-03-29T06:45:26.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:45:26.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:45:26.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:45:26.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:45:26.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:45:26.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:45:26.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:45:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:45:26.323 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:45:26.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:45:26.324 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:45:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:45:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:45:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:45:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:45:26.357 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:26.355+0000 7f01fbd197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:26.360 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:26.359+0000 7f01fbd197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:26.361 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:26.359+0000 7f01fbd197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:26.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:26.895 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:26.891+0000 7f01fbd197c0 -1 Falling back to public interface 2024-03-29T06:45:26.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:27.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:27.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:27.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:45:27.926 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:45:27.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:27.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:28.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:29.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:29.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:29.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:45:29.241 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:45:29.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:29.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:29.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:29.856 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:29.851+0000 7f01fbd197c0 -1 osd.1 63 log_to_monitors true 2024-03-29T06:45:30.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:30.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:30.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:45:30.552 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:45:30.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:30.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:30.861 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 67 up_thru 22 down_at 64 last_clean_interval [10,63) [v2:127.0.0.1:6810/301731603,v1:127.0.0.1:6811/301731603] [v2:127.0.0.1:6812/301731603,v1:127.0.0.1:6813/301731603] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:45:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:45:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:45:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:45:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:45:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:45:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:45:30.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:45:30.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:45:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:45:30.863 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:45:30.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:45:30.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:45:30.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:45:30.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:45:30.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:45:30.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:45:30.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:45:30.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:45:30.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:45:31.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:45:31.383 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:45:31.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:45:31.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:45:31.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:45:31.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-03-29T06:45:31.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-03-29T06:45:31.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-03-29T06:45:31.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:45:31.758 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:45:32.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-29T06:45:32.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-29T06:45:32.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-287762808834' 2024-03-29T06:45:32.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:45:32.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-03-29T06:45:32.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:45:32.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:45:32.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-03-29T06:45:32.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:45:32.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-03-29T06:45:32.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-03-29T06:45:32.045 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836513 2024-03-29T06:45:32.045 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:45:32.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-03-29T06:45:32.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:45:33.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:45:33.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:45:33.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-03-29T06:45:33.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:45:33.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-03-29T06:45:33.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:45:33.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:45:33.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-03-29T06:45:33.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:45:33.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-29T06:45:33.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-03-29T06:45:33.808 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 287762808834 2024-03-29T06:45:33.808 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:45:34.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-29T06:45:34.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:45:34.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:45:34.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:45:34.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:45:34.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:45:34.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:45:34.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:45:34.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:45:34.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:45:34.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:45:34.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:45:34.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:45:34.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:45:34.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:45:34.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:45:35.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:45:35.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:45:35.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:45:35.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:45:35.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:45:35.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:45:35.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:45:35.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:45:35.273 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:45:35.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:45:35.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:45:35.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:45:35.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:45:35.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:45:35.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:45:35.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:45:35.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:45:35.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:45:35.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:45:35.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:45:35.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:45:35.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:45:35.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:45:35.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:45:35.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:45:35.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:45:35.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:45:36.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:45:36.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:45:36.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:45:36.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:45:36.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:45:36.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:45:36.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:45:36.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:45:36.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:45:36.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:45:36.494 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:45:36.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:45:36.495 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:45:36.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:45:36.788 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:45:36.788 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:45:36.788 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:45:37.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:45:37.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:45:37.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:45:37.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:37.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-29T06:45:37.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-03-29T06:45:37.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:45:37.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:45:37.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-03-29T06:45:37.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:45:37.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:45:37.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:45:37.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:45:37.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:45:37.143 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:45:37.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:45:37.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:45:37.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:45:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:45:40.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:45:40.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:45:40.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:45:40.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:45:40.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:45:40.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:45:40.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:45:40.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:45:40.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:45:40.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:45:40.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:45:40.827 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:45:40.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:45:40.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:45:40.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:45:40.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:45:40.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:45:40.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:45:40.860 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:40.854+0000 7f5d681417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:40.862 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:40.858+0000 7f5d681417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:40.864 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:40.862+0000 7f5d681417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:41.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:45:41.399 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:41.394+0000 7f5d681417c0 -1 Falling back to public interface 2024-03-29T06:45:41.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:42.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:42.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:42.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:45:42.464 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:45:42.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:42.465 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:45:42.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:43.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:43.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:43.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:45:43.784 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:45:43.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:43.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:45:44.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:44.327 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:44.322+0000 7f5d681417c0 -1 osd.0 69 log_to_monitors true 2024-03-29T06:45:45.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:45.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:45.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:45:45.107 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:45:45.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:45.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [5,69) [v2:127.0.0.1:6802/3919729262,v1:127.0.0.1:6803/3919729262] [v2:127.0.0.1:6804/3919729262,v1:127.0.0.1:6805/3919729262] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:45:45.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:45:45.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:45:45.763 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:45:45.763 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:45:45.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:45:45.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:45:45.764 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:45:45.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:45:45.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:45:45.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:45:45.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:45:45.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:45:45.878 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:45:46.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:45:46.381 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:45:46.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:45:46.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:45:46.381 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:45:46.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-29T06:45:46.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-29T06:45:46.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-313532612610' 2024-03-29T06:45:46.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:45:46.530 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:45:46.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808838 2024-03-29T06:45:46.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808838 2024-03-29T06:45:46.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-313532612610 1-287762808838' 2024-03-29T06:45:46.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:45:46.682 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-313532612610 2024-03-29T06:45:46.682 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:45:46.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:45:46.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-313532612610 2024-03-29T06:45:46.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:45:46.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-29T06:45:46.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 313532612610' 2024-03-29T06:45:46.685 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 313532612610 2024-03-29T06:45:46.685 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:45:47.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 313532612610 2024-03-29T06:45:47.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:45:48.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:45:48.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:45:48.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 313532612610 2024-03-29T06:45:48.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:45:49.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T06:45:49.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:45:49.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-03-29T06:45:49.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:45:49.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808838 2024-03-29T06:45:49.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:45:49.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:45:49.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808838 2024-03-29T06:45:49.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:45:49.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808838 2024-03-29T06:45:49.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808838' 2024-03-29T06:45:49.680 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 287762808838 2024-03-29T06:45:49.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:45:49.979 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808838 -lt 287762808838 2024-03-29T06:45:49.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:45:49.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:45:49.980 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:45:50.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:45:50.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:45:50.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:45:50.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:45:50.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:45:50.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:45:50.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:45:50.371 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:45:50.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:45:50.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:45:50.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:45:50.665 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:45:51.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:45:51.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:45:51.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:45:51.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:45:51.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-29T06:45:51.005 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-03-29T06:45:51.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:45:51.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:45:51.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:45:51.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:45:51.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:45:51.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:45:52.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:45:52.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-03-29T06:45:52.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:45:52.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:45:52.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:45:52.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:45:52.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:45:52.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-03-29T06:45:53.977 INFO:tasks.workunit.client.0.smithi123.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-03-29T06:45:54.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:45:54.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:45:54.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:45:54.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:45:54.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:45:54.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:45:54.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:45:54.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:45:54.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:45:54.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:45:54.524 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:45:54.525 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:45:54.525 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:45:54.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:45:54.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:45:54.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:45:54.527 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:45:54.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:45:54.528 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:45:54.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:45:54.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:45:54.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:45:54.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:45:54.560 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:54.562+0000 7fdb0f64c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:54.563 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:54.562+0000 7fdb0f64c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:54.564 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:54.566+0000 7fdb0f64c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:45:54.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:45:54.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:45:54.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:45:54.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:45:54.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:45:54.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:54.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:45:54.863 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:45:54.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:54.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:55.118 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:55.118+0000 7fdb0f64c7c0 -1 Falling back to public interface 2024-03-29T06:45:55.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:56.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:56.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:56.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:45:56.188 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:45:56.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:56.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:56.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:57.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:57.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:57.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:45:57.558 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:45:57.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:57.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:57.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:45:58.061 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:45:58.062+0000 7fdb0f64c7c0 -1 osd.1 74 log_to_monitors true 2024-03-29T06:45:58.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:45:58.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:45:58.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:45:58.861 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:45:58.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:45:58.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:45:59.413 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [67,74) [v2:127.0.0.1:6810/2496077168,v1:127.0.0.1:6811/2496077168] [v2:127.0.0.1:6812/2496077168,v1:127.0.0.1:6813/2496077168] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:45:59.414 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:45:59.415 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:45:59.415 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:45:59.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:45:59.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:45:59.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:45:59.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:45:59.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:45:59.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:45:59.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:45:59.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:45:59.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:45:59.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:45:59.975 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:45:59.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:45:59.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:45:59.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:46:00.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612614 2024-03-29T06:46:00.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612614 2024-03-29T06:46:00.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-313532612614' 2024-03-29T06:46:00.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:46:00.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:46:00.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-03-29T06:46:00.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-03-29T06:46:00.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-313532612614 1-335007449090' 2024-03-29T06:46:00.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:00.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-313532612614 2024-03-29T06:46:00.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:00.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:46:00.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-313532612614 2024-03-29T06:46:00.567 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:00.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612614 2024-03-29T06:46:00.568 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 313532612614' 2024-03-29T06:46:00.568 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 313532612614 2024-03-29T06:46:00.569 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:46:00.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612614 -lt 313532612614 2024-03-29T06:46:00.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:00.878 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-335007449090 2024-03-29T06:46:00.878 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:00.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:46:00.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-335007449090 2024-03-29T06:46:00.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:00.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-03-29T06:46:00.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2024-03-29T06:46:00.883 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 335007449090 2024-03-29T06:46:00.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:46:01.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 335007449090 2024-03-29T06:46:01.204 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:46:02.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:46:02.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:46:02.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449090 -lt 335007449090 2024-03-29T06:46:02.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:46:02.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:02.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:02.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:46:02.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:02.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:02.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:02.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:02.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:02.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:02.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:03.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:03.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:03.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:03.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:03.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:03.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:46:03.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:46:03.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:46:03.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:03.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:46:03.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:03.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:03.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:03.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:03.726 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:03.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:03.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:03.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:04.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:04.095 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:04.095 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:04.095 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:04.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:04.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:46:04.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:46:04.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:46:04.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:46:04.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:46:04.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:46:04.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:46:04.452 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:46:04.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:46:04.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:46:04.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:46:04.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:04.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:46:05.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:05.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:05.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:05.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:05.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:05.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:05.005 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:05.005 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:05.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:05.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:05.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:05.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:05.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:05.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:46:05.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:46:05.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:46:05.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:46:05.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:46:05.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:46:05.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:46:05.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:46:06.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:46:06.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:46:06.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:46:06.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:06.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:46:06.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:06.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:06.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:06.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:06.571 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:06.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:06.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:06.572 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:06.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:46:06.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:06.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:06.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:07.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:46:07.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:46:07.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:46:07.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:46:07.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-29T06:46:07.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-03-29T06:46:07.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:46:07.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:46:07.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:46:07.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:46:07.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:46:07.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:46:07.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:46:07.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:46:07.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:46:08.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:46:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-29T06:46:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:46:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:46:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:46:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:46:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:46:08.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-29T06:46:10.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:46:10.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:46:10.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:46:10.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:46:10.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:46:10.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:46:10.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:46:10.640 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:46:10.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:46:10.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:46:10.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:46:10.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:46:10.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:46:10.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:46:10.675 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:10.674+0000 7fe73da887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:10.678 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:10.678+0000 7fe73da887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:10.679 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:10.678+0000 7fe73da887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:10.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:46:10.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:46:10.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:46:10.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:46:10.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:46:10.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:10.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:46:10.958 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:46:10.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:10.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:11.214 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:11.214+0000 7fe73da887c0 -1 Falling back to public interface 2024-03-29T06:46:11.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:12.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:12.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:12.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:46:12.310 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:46:12.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:12.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:12.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:13.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:13.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:13.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:46:13.703 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:46:13.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:13.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:13.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:14.312 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:14.314+0000 7fe73da887c0 -1 osd.0 80 log_to_monitors true 2024-03-29T06:46:14.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:14.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:14.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:46:14.923 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:46:14.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:14.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 84 up_thru 79 down_at 81 last_clean_interval [73,80) [v2:127.0.0.1:6802/1421414635,v1:127.0.0.1:6803/1421414635] [v2:127.0.0.1:6804/1421414635,v1:127.0.0.1:6805/1421414635] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:46:15.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:46:15.336 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:46:15.336 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:46:15.336 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:46:15.336 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:46:15.336 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:46:15.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:46:15.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:46:15.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:46:15.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:46:15.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:46:15.451 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:46:15.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:46:15.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:46:15.902 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:46:15.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:46:15.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:46:15.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:46:16.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-29T06:46:16.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-29T06:46:16.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866' 2024-03-29T06:46:16.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:46:16.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:46:16.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449094 2024-03-29T06:46:16.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449094 2024-03-29T06:46:16.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866 1-335007449094' 2024-03-29T06:46:16.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:16.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252866 2024-03-29T06:46:16.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:16.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:46:16.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252866 2024-03-29T06:46:16.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:16.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-29T06:46:16.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-03-29T06:46:16.596 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 360777252866 2024-03-29T06:46:16.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:46:16.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 360777252866 2024-03-29T06:46:16.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:46:17.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:46:17.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:46:18.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 360777252866 2024-03-29T06:46:18.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:46:19.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T06:46:19.288 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:46:19.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-03-29T06:46:19.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:19.606 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-335007449094 2024-03-29T06:46:19.606 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:19.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:46:19.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-335007449094 2024-03-29T06:46:19.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:19.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449094 2024-03-29T06:46:19.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 335007449094' 2024-03-29T06:46:19.610 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 335007449094 2024-03-29T06:46:19.611 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:46:19.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449094 -lt 335007449094 2024-03-29T06:46:19.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:46:19.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:19.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:20.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:46:20.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:20.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:20.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:20.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:20.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:20.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:20.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:20.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:46:20.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:20.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:20.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:20.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:46:20.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:46:20.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:46:20.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:46:20.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-29T06:46:20.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:46:20.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:46:20.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:46:21.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-03-29T06:46:24.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:46:24.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:46:24.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:46:24.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:46:24.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:46:24.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:46:24.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:46:24.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:46:24.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:46:24.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:46:24.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:46:24.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:46:24.031 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:46:24.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:46:24.032 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:46:24.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:46:24.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:46:24.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:46:24.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:46:24.065 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:24.065+0000 7f124e85d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:24.068 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:24.069+0000 7f124e85d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:24.069 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:24.069+0000 7f124e85d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:24.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:46:24.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:46:24.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:46:24.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:46:24.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:46:24.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:24.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:46:24.385 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:46:24.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:24.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:24.603 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:24.601+0000 7f124e85d7c0 -1 Falling back to public interface 2024-03-29T06:46:24.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:25.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:25.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:25.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:46:25.749 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:46:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:25.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:26.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:27.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:27.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:27.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:46:27.065 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:46:27.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:27.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:27.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:27.513 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:27.513+0000 7f124e85d7c0 -1 osd.1 86 log_to_monitors true 2024-03-29T06:46:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:46:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:46:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:29.029 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 90 up_thru 84 down_at 87 last_clean_interval [78,86) [v2:127.0.0.1:6810/3636177092,v1:127.0.0.1:6811/3636177092] [v2:127.0.0.1:6812/3636177092,v1:127.0.0.1:6813/3636177092] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:46:29.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:46:29.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:46:29.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:46:29.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:46:29.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:46:29.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:46:29.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:46:29.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:46:29.031 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:46:29.031 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:46:29.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:46:29.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:46:29.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:46:29.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:46:29.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:46:29.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:46:29.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:46:29.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:46:29.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:46:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:46:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:46:29.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:46:29.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:46:29.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:46:29.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252869 2024-03-29T06:46:29.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252869 2024-03-29T06:46:29.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869' 2024-03-29T06:46:29.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:46:29.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:46:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-03-29T06:46:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-03-29T06:46:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869 1-386547056642' 2024-03-29T06:46:29.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:29.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252869 2024-03-29T06:46:29.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:29.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:46:29.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252869 2024-03-29T06:46:29.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:29.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252869 2024-03-29T06:46:29.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-03-29T06:46:29.843 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 360777252869 2024-03-29T06:46:29.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:46:30.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252868 -lt 360777252869 2024-03-29T06:46:30.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:46:31.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:46:31.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:46:31.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252870 -lt 360777252869 2024-03-29T06:46:31.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:31.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:31.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-03-29T06:46:31.458 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:46:31.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-03-29T06:46:31.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:31.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-03-29T06:46:31.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-03-29T06:46:31.460 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 386547056642 2024-03-29T06:46:31.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:46:31.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-03-29T06:46:31.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:46:31.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:31.755 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:32.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:46:32.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:32.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:32.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:32.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:32.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:32.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:32.135 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:32.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:32.416 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:32.416 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:32.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:32.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:32.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:46:32.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:46:32.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:46:32.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:32.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:46:32.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:32.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:32.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:32.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:32.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:32.865 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:32.866 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:32.866 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:33.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:33.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:33.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:33.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:46:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:46:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:46:33.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:46:33.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:46:33.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:46:33.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:46:33.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:46:33.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:46:33.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:46:33.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:46:33.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:33.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:46:34.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:34.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:34.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:34.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:34.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:34.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:34.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:34.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:34.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:34.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:34.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:34.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:34.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:34.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:46:34.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:46:34.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:46:34.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:46:34.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:46:34.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:46:34.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:46:34.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:46:35.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:46:35.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:46:35.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:46:35.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:35.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:46:35.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:35.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:35.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:35.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:35.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:35.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:35.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:35.471 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:35.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:35.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:35.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:35.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:46:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:46:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:46:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:46:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:46:36.131 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:46:36.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:46:36.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:46:36.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:46:36.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:46:36.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-29T06:46:36.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:36.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-29T06:46:37.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:37.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:37.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:37.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:37.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:37.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:37.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:37.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:37.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:46:37.589 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:37.589 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:37.589 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:37.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:46:37.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:46:37.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:46:37.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:46:37.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-29T06:46:37.950 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:46:37.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:46:37.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:46:37.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:46:37.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:46:37.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:46:37.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:46:37.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:46:37.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:46:38.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:46:38.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-29T06:46:38.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:46:38.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:46:38.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:46:38.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:46:38.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:46:38.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-29T06:46:40.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:46:40.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:46:40.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:46:40.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:46:40.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:46:40.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:46:40.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:46:40.844 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:46:40.845 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:46:40.845 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:40.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:40.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:46:40.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:46:40.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:46:40.847 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:46:40.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:46:40.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:46:40.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:46:40.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:46:40.878 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:40.877+0000 7fac044917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:40.881 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:40.881+0000 7fac044917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:40.882 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:40.881+0000 7fac044917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:41.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:46:41.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:46:41.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:46:41.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:46:41.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:46:41.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:41.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:46:41.143 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:46:41.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:41.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:41.415 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:41.413+0000 7fac044917c0 -1 Falling back to public interface 2024-03-29T06:46:41.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:42.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:42.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:42.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:46:42.453 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:46:42.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:42.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:42.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:43.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:43.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:43.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:46:43.763 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:46:43.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:43.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:44.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:44.088 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:44.089+0000 7fac044917c0 -1 osd.0 91 log_to_monitors true 2024-03-29T06:46:45.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:45.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:45.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:46:45.077 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:46:45.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:45.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 95 up_thru 90 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6802/1016897678,v1:127.0.0.1:6803/1016897678] [v2:127.0.0.1:6804/1016897678,v1:127.0.0.1:6805/1016897678] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:46:45.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:46:45.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:46:45.556 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:46:45.556 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:46:45.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:46:45.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:46:45.557 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:46:45.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:46:45.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:46:45.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:46:45.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:46:45.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:46:45.677 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:46:46.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:46:46.020 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:46:46.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:46:46.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:46:46.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:46:46.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-03-29T06:46:46.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-03-29T06:46:46.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122' 2024-03-29T06:46:46.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:46:46.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:46:46.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056646 2024-03-29T06:46:46.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056646 2024-03-29T06:46:46.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122 1-386547056646' 2024-03-29T06:46:46.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:46.586 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893122 2024-03-29T06:46:46.586 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:46.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:46:46.588 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893122 2024-03-29T06:46:46.588 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:46.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-03-29T06:46:46.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2024-03-29T06:46:46.589 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 408021893122 2024-03-29T06:46:46.590 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:46:47.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893122 -lt 408021893122 2024-03-29T06:46:47.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:46:47.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056646 2024-03-29T06:46:47.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:46:47.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:46:47.018 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056646 2024-03-29T06:46:47.018 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:46:47.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056646 2024-03-29T06:46:47.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056646' 2024-03-29T06:46:47.019 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 386547056646 2024-03-29T06:46:47.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:46:47.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056645 -lt 386547056646 2024-03-29T06:46:47.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:46:48.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:46:48.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:46:48.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056647 -lt 386547056646 2024-03-29T06:46:48.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:46:48.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:48.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:49.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:46:49.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:49.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:49.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:49.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:49.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:49.012 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:49.013 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:49.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:49.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:49.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:49.322 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:49.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:49.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:46:49.675 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:46:49.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:46:49.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:49.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:46:49.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:49.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:49.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:49.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:49.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:49.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:49.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:49.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:50.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:46:50.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:50.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:50.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:50.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:46:50.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:46:50.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:46:50.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:46:50.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:46:50.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:46:50.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:46:50.413 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:46:50.413 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:46:50.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:46:50.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:46:50.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:46:50.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:46:50.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:46:50.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:46:50.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:46:50.958 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:46:50.958 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:46:50.958 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:46:50.958 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:46:50.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:46:50.959 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:46:51.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:46:51.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:46:51.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:46:51.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:46:51.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:46:51.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:46:51.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:46:51.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-03-29T06:46:51.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:46:51.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-29T06:46:51.629 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:46:51.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:46:51.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:46:52.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:46:52.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-29T06:46:52.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:46:52.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:46:52.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:46:52.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:46:52.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:46:52.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-29T06:46:54.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:46:54.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:46:54.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:46:54.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:46:54.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:46:54.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:46:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:46:54.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:46:54.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:46:54.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:46:54.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:46:54.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:46:54.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:46:54.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:46:54.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:46:54.759 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:46:54.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:46:54.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:46:54.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:46:54.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:46:54.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:46:54.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:46:54.791 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:54.789+0000 7f2a309397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:54.794 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:54.793+0000 7f2a309397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:54.795 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:54.793+0000 7f2a309397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:46:55.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:46:55.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:46:55.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:46:55.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:46:55.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:46:55.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:55.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:46:55.074 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:46:55.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:55.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:55.338 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:55.337+0000 7f2a309397c0 -1 Falling back to public interface 2024-03-29T06:46:55.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:56.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:56.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:56.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:46:56.372 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:46:56.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:56.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:56.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:57.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:57.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:57.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:46:57.721 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:46:57.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:57.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:58.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:46:58.217 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:46:58.217+0000 7f2a309397c0 -1 osd.1 97 log_to_monitors true 2024-03-29T06:46:59.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:46:59.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:46:59.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:46:59.029 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:46:59.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:46:59.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:46:59.538 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/1612561233,v1:127.0.0.1:6811/1612561233] [v2:127.0.0.1:6812/1612561233,v1:127.0.0.1:6813/1612561233] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:46:59.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:46:59.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:46:59.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:46:59.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:46:59.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:46:59.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:46:59.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:46:59.539 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:46:59.539 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:46:59.539 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:46:59.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:46:59.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:46:59.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:46:59.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:46:59.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:46:59.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:46:59.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:46:59.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:46:59.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:47:00.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:47:00.120 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:47:00.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:47:00.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:00.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:47:00.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893126 2024-03-29T06:47:00.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893126 2024-03-29T06:47:00.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893126' 2024-03-29T06:47:00.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:00.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:47:00.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-03-29T06:47:00.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-03-29T06:47:00.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893126 1-433791696898' 2024-03-29T06:47:00.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:00.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893126 2024-03-29T06:47:00.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:00.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:47:00.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893126 2024-03-29T06:47:00.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:00.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893126 2024-03-29T06:47:00.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893126' 2024-03-29T06:47:00.435 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 408021893126 2024-03-29T06:47:00.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:00.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893125 -lt 408021893126 2024-03-29T06:47:00.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:47:01.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:47:01.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:02.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893125 -lt 408021893126 2024-03-29T06:47:02.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:47:03.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T06:47:03.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:03.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893126 -lt 408021893126 2024-03-29T06:47:03.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:03.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696898 2024-03-29T06:47:03.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:03.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:47:03.393 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696898 2024-03-29T06:47:03.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:03.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-03-29T06:47:03.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2024-03-29T06:47:03.395 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 433791696898 2024-03-29T06:47:03.395 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:47:03.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-03-29T06:47:03.714 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:47:03.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:03.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:04.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:47:04.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:04.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:04.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:04.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:04.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:04.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:04.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:04.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:47:04.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:04.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:04.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:04.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:47:04.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:47:04.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:47:04.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-03-29T06:47:04.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:47:04.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-29T06:47:04.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-03-29T06:47:04.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:47:04.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:47:04.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-03-29T06:47:04.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-03-29T06:47:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-29T06:47:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-29T06:47:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:47:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:47:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:47:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:47:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:47:05.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-29T06:47:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:47:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:47:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:47:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:47:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:47:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:47:07.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:47:07.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:47:07.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:47:07.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:47:07.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:47:07.861 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:47:07.861 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:47:07.861 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:47:07.861 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:47:07.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:47:07.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:47:07.863 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:47:07.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:47:07.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:47:07.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:47:07.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:47:07.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:47:07.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:47:07.898 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:07.896+0000 7fd3f0fc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:07.900 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:07.900+0000 7fd3f0fc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:07.901 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:07.900+0000 7fd3f0fc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:47:08.173 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:47:08.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:47:08.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:08.431 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:08.428+0000 7fd3f0fc57c0 -1 Falling back to public interface 2024-03-29T06:47:08.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:47:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:47:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:09.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:47:09.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:10.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:10.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:10.787 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:47:10.788 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:47:10.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:10.788 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:47:11.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:11.307 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:11.304+0000 7fd3f0fc57c0 -1 osd.0 102 log_to_monitors true 2024-03-29T06:47:12.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:12.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:12.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:47:12.096 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:47:12.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:12.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 106 up_thru 101 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6802/2883014153,v1:127.0.0.1:6803/2883014153] [v2:127.0.0.1:6804/2883014153,v1:127.0.0.1:6805/2883014153] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:47:12.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:47:12.773 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:47:12.773 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:47:12.773 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:47:12.773 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:47:12.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:47:12.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:47:12.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:47:12.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:47:12.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:47:12.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:47:12.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:47:12.899 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:47:13.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:47:13.317 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:47:13.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:47:13.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:13.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:47:13.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-03-29T06:47:13.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-03-29T06:47:13.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378' 2024-03-29T06:47:13.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:13.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:47:13.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696902 2024-03-29T06:47:13.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696902 2024-03-29T06:47:13.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378 1-433791696902' 2024-03-29T06:47:13.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:13.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533378 2024-03-29T06:47:13.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:13.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:47:13.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533378 2024-03-29T06:47:13.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-03-29T06:47:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2024-03-29T06:47:13.817 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 455266533378 2024-03-29T06:47:13.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:14.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 455266533378 2024-03-29T06:47:14.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:47:15.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:47:15.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-03-29T06:47:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:15.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696902 2024-03-29T06:47:15.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:15.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:47:15.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696902 2024-03-29T06:47:15.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:15.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696902 2024-03-29T06:47:15.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696902' 2024-03-29T06:47:15.450 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 433791696902 2024-03-29T06:47:15.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:47:15.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696902 -lt 433791696902 2024-03-29T06:47:15.765 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:47:15.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:15.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:16.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:47:16.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:16.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:16.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:16.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:16.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:16.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:16.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:16.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:47:16.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:16.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:16.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:47:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-29T06:47:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:47:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-29T06:47:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:47:16.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:16.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:16.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:16.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:16.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:16.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:16.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:16.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:17.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:47:17.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:17.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:17.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:17.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:47:17.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:47:17.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:47:17.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:47:17.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:47:17.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:47:17.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:47:17.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:47:17.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:47:17.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:47:17.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:47:17.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:47:17.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:17.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:47:18.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:18.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:18.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:18.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:18.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:18.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:18.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:18.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:18.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:47:18.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:18.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:18.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:18.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:47:18.870 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:47:19.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:47:19.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:47:19.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:47:19.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:19.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:47:19.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:19.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:19.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:19.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:19.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:19.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:19.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:19.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:19.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:47:19.929 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:19.929 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:19.930 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:20.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-29T06:47:20.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-29T06:47:20.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:47:20.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:47:20.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:47:20.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:47:20.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:47:20.297 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:47:20.297 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:47:20.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:47:20.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:47:20.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-29T06:47:20.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:20.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-29T06:47:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:21.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:21.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:21.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:21.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:21.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:21.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:21.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:21.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:47:21.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:21.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:21.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:22.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:47:22.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:47:22.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:47:22.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:47:22.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:47:22.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:47:22.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:47:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:47:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-29T06:47:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:47:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:47:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:47:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:47:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:47:23.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:47:25.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:47:25.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:47:25.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:47:25.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:47:25.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:47:25.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:25.707 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:47:25.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:47:25.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:47:25.709 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:47:25.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:47:25.710 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:47:25.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:47:25.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:47:25.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:47:25.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:47:25.742 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:25.740+0000 7f38f50797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:25.746 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:25.744+0000 7f38f50797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:25.747 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:25.744+0000 7f38f50797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:26.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:26.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:26.283 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:26.280+0000 7f38f50797c0 -1 Falling back to public interface 2024-03-29T06:47:26.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:27.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:27.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:27.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:47:27.340 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:47:27.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:27.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:27.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:47:28.649 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:47:28.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:28.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:29.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:29.594 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:29.592+0000 7f38f50797c0 -1 osd.1 108 log_to_monitors true 2024-03-29T06:47:30.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:30.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:30.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:47:30.115 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:47:30.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:30.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:30.505 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 112 up_thru 106 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/3702477745,v1:127.0.0.1:6811/3702477745] [v2:127.0.0.1:6812/3702477745,v1:127.0.0.1:6813/3702477745] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:47:30.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:47:30.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:47:30.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:47:30.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:47:30.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:47:30.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:47:30.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:47:30.507 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:47:30.507 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:47:30.507 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:47:30.507 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:47:30.508 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:47:30.508 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:47:30.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:47:30.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:47:30.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:47:30.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:47:30.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:47:30.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:47:31.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:47:31.067 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:47:31.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:47:31.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:31.067 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:47:31.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533382 2024-03-29T06:47:31.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533382 2024-03-29T06:47:31.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533382' 2024-03-29T06:47:31.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:31.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:47:31.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-03-29T06:47:31.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-03-29T06:47:31.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533382 1-481036337154' 2024-03-29T06:47:31.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:31.381 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533382 2024-03-29T06:47:31.381 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:31.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:47:31.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533382 2024-03-29T06:47:31.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:31.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533382 2024-03-29T06:47:31.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533382' 2024-03-29T06:47:31.384 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 455266533382 2024-03-29T06:47:31.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:31.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533381 -lt 455266533382 2024-03-29T06:47:31.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:47:32.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:47:32.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:33.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533383 -lt 455266533382 2024-03-29T06:47:33.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:33.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-481036337154 2024-03-29T06:47:33.146 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:33.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:47:33.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-481036337154 2024-03-29T06:47:33.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:33.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-03-29T06:47:33.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2024-03-29T06:47:33.154 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 481036337154 2024-03-29T06:47:33.154 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:47:33.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337154 -lt 481036337154 2024-03-29T06:47:33.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:47:33.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:33.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:33.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:47:33.826 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:33.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:33.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:33.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:33.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:33.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:33.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:34.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:47:34.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:34.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:34.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:47:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T06:47:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:47:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T06:47:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:34.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:47:34.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:34.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:34.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:34.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:34.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:34.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:34.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:34.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:34.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:47:34.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:34.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:34.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:35.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:47:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-29T06:47:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:47:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:47:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:47:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:47:35.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:47:35.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:47:35.243 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:47:35.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:47:35.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:47:35.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:47:35.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:35.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:47:35.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:35.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:35.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:35.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:35.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:35.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:35.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:35.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:36.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T06:47:36.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:36.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:36.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:36.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-03-29T06:47:36.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-29T06:47:36.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:47:36.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:47:36.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:47:36.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:47:36.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:47:36.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:47:36.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:47:36.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:47:36.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:47:36.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:47:36.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:36.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:47:37.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:37.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:37.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:37.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:37.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:37.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:37.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:37.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:37.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:47:37.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:37.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:37.575 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:37.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:47:37.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 2 2024-03-29T06:47:37.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:47:37.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:47:37.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:37.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:47:38.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:38.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:38.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:38.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:38.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:38.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:38.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:47:38.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:38.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:38.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:47:38.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:47:38.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:47:39.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:47:39.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-29T06:47:39.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:47:39.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:47:39.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:47:39.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:47:39.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:47:39.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-29T06:47:41.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:47:41.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:47:41.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:47:41.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:47:41.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:47:41.711 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:47:41.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:47:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:47:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:47:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:47:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:47:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:47:41.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:47:41.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:47:41.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:47:41.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:47:41.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:47:41.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:47:41.748 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:41.744+0000 7f849046b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:41.750 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:41.748+0000 7f849046b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:41.751 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:41.748+0000 7f849046b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:47:42.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:42.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:42.303 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:42.300+0000 7f849046b7c0 -1 Falling back to public interface 2024-03-29T06:47:42.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:43.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:43.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:43.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:47:43.376 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:47:43.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:43.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:43.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:47:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:47:44.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:44.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:45.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:45.253 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:45.251+0000 7f849046b7c0 -1 osd.1 113 log_to_monitors true 2024-03-29T06:47:46.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:46.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:46.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:47:46.099 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:47:46.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:46.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:46.614 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 117 up_thru 112 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/1779613736,v1:127.0.0.1:6811/1779613736] [v2:127.0.0.1:6812/1779613736,v1:127.0.0.1:6813/1779613736] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:47:46.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:47:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:47:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:47:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:47:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:47:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:47:46.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:47:46.615 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:47:46.616 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:47:46.616 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:47:46.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:47:46.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:47:46.616 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:47:46.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:47:46.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:47:46.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:47:46.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:47:46.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:47:46.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:47:47.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:47:47.163 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:47:47.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:47:47.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:47.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:47:47.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533387 2024-03-29T06:47:47.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533387 2024-03-29T06:47:47.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533387' 2024-03-29T06:47:47.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:47:47.391 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:47:47.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-03-29T06:47:47.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-03-29T06:47:47.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533387 1-502511173634' 2024-03-29T06:47:47.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:47.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533387 2024-03-29T06:47:47.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:47.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:47:47.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533387 2024-03-29T06:47:47.729 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:47.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533387 2024-03-29T06:47:47.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533387' 2024-03-29T06:47:47.732 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 455266533387 2024-03-29T06:47:47.732 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:48.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533387 2024-03-29T06:47:48.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:47:49.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:47:49.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:47:49.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533387 -lt 455266533387 2024-03-29T06:47:49.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:47:49.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-502511173634 2024-03-29T06:47:49.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:47:49.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:47:49.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-502511173634 2024-03-29T06:47:49.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:47:49.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-03-29T06:47:49.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-03-29T06:47:49.555 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 502511173634 2024-03-29T06:47:49.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:47:49.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-03-29T06:47:49.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:47:49.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:49.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:50.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:47:50.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:50.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:50.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:50.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:50.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:50.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:50.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:50.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T06:47:50.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:50.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:50.565 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:50.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-03-29T06:47:50.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-29T06:47:50.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:47:50.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-29T06:47:50.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:50.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:47:51.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:51.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:51.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:51.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:51.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:51.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:51.045 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:51.045 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:51.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T06:47:51.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:51.338 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:51.338 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-03-29T06:47:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-29T06:47:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:47:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:47:51.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:47:51.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:47:51.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:47:51.690 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:47:51.690 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:47:52.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:47:52.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:47:52.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:47:52.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:47:52.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:47:52.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:47:52.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:47:52.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:47:52.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:47:52.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:47:52.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:47:52.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:47:52.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:47:52.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:47:52.576 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:47:52.576 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:47:52.577 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:47:52.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:47:52.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:47:52.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:47:52.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-29T06:47:52.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:47:52.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-29T06:47:52.952 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-03-29T06:47:52.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:47:52.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:47:52.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:47:52.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:47:52.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:47:52.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:47:52.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:47:52.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:47:52.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:47:52.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:47:53.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:47:53.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-29T06:47:53.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:47:53.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:47:53.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:47:53.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:47:53.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:47:53.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:47:55.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:47:55.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:47:55.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:47:55.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:47:55.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:47:55.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:47:55.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:47:55.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:47:55.915 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:47:55.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:47:55.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:47:55.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:47:55.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:47:55.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:47:55.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:47:55.948 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:55.943+0000 7f5117d657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:55.950 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:55.947+0000 7f5117d657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:55.952 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:55.951+0000 7f5117d657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:47:56.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:47:56.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:47:56.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:47:56.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:47:56.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:47:56.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:56.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:47:56.244 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:47:56.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:56.244 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:56.502 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:56.499+0000 7f5117d657c0 -1 Falling back to public interface 2024-03-29T06:47:56.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:57.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:57.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:57.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:47:57.556 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:47:57.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:57.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:57.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:58.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:47:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:47:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:47:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:47:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:47:58.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:47:59.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:47:59.269 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:47:59.267+0000 7f5117d657c0 -1 osd.1 119 log_to_monitors true 2024-03-29T06:48:00.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:00.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:00.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:48:00.238 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:48:00.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:00.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:00.792 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 123 up_thru 117 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/172882581,v1:127.0.0.1:6811/172882581] [v2:127.0.0.1:6812/172882581,v1:127.0.0.1:6813/172882581] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:48:00.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:48:00.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:48:00.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:48:00.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:48:00.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:48:00.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:48:00.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:48:00.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:48:00.794 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:48:00.794 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:48:00.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:48:00.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:48:00.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:48:00.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:48:00.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:48:00.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:48:00.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:48:00.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:48:00.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:48:01.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:48:01.514 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:48:01.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:48:01.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:01.514 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:48:01.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533390 2024-03-29T06:48:01.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533390 2024-03-29T06:48:01.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533390' 2024-03-29T06:48:01.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:01.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:48:01.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-03-29T06:48:01.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-03-29T06:48:01.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533390 1-528280977410' 2024-03-29T06:48:01.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:01.802 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533390 2024-03-29T06:48:01.802 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:01.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:48:01.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533390 2024-03-29T06:48:01.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:01.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533390 2024-03-29T06:48:01.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533390' 2024-03-29T06:48:01.805 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 455266533390 2024-03-29T06:48:01.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:02.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533389 -lt 455266533390 2024-03-29T06:48:02.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:48:03.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:48:03.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:03.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533391 -lt 455266533390 2024-03-29T06:48:03.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:03.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-528280977410 2024-03-29T06:48:03.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:03.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:48:03.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-528280977410 2024-03-29T06:48:03.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:03.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-03-29T06:48:03.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-03-29T06:48:03.440 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 528280977410 2024-03-29T06:48:03.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:48:03.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-03-29T06:48:03.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:48:03.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:03.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:04.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:48:04.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:04.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:04.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:04.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:04.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:04.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:04.181 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:04.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:48:04.515 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:04.515 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:04.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:04.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:48:04.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:48:04.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:48:04.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-03-29T06:48:04.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-29T06:48:05.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:48:05.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:48:05.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:48:05.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:48:05.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:48:05.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:48:05.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:48:05.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:48:05.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:48:05.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:48:05.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:48:05.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:48:05.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:48:05.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:48:05.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:48:05.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:48:05.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:48:05.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:48:05.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:48:05.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:48:05.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:48:05.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:48:05.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:48:05.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:48:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:48:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:48:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:48:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:48:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:48:08.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:48:08.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:48:08.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:48:08.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:48:08.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:48:08.123 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:48:08.123 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:48:08.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:48:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:48:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:48:08.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:48:08.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:48:08.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:48:08.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:48:08.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:48:08.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:48:08.159 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:08.155+0000 7f0630b337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:08.161 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:08.159+0000 7f0630b337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:08.163 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:08.159+0000 7f0630b337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:08.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:08.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:08.726 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:08.723+0000 7f0630b337c0 -1 Falling back to public interface 2024-03-29T06:48:09.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:48:10.078 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:48:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:10.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:10.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:11.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:11.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:11.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:48:11.406 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:48:11.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:11.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:11.563+0000 7f0630b337c0 -1 osd.1 124 log_to_monitors true 2024-03-29T06:48:11.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:12.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:12.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:12.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:48:12.720 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:48:12.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:12.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:13.307 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 128 up_thru 128 down_at 125 last_clean_interval [123,124) [v2:127.0.0.1:6810/1786675587,v1:127.0.0.1:6811/1786675587] [v2:127.0.0.1:6812/1786675587,v1:127.0.0.1:6813/1786675587] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:48:13.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:48:13.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:48:13.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:48:13.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:48:13.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:48:13.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:48:13.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:48:13.309 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:48:13.309 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:48:13.309 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:48:13.309 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:48:13.309 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:48:13.309 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:48:13.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:48:13.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:48:13.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:48:13.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:48:13.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:48:13.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:48:13.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:48:13.769 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:48:13.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:48:13.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:13.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:48:13.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533394 2024-03-29T06:48:13.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533394 2024-03-29T06:48:13.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533394' 2024-03-29T06:48:13.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:13.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:48:14.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=549755813890 2024-03-29T06:48:14.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 549755813890 2024-03-29T06:48:14.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533394 1-549755813890' 2024-03-29T06:48:14.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:14.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533394 2024-03-29T06:48:14.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:14.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:48:14.170 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533394 2024-03-29T06:48:14.170 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:14.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533394 2024-03-29T06:48:14.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533394' 2024-03-29T06:48:14.172 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 455266533394 2024-03-29T06:48:14.172 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:14.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533394 -lt 455266533394 2024-03-29T06:48:14.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:14.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:14.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-549755813890 2024-03-29T06:48:14.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:48:14.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-549755813890 2024-03-29T06:48:14.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:14.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=549755813890 2024-03-29T06:48:14.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 549755813890' 2024-03-29T06:48:14.505 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 549755813890 2024-03-29T06:48:14.505 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:48:14.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 549755813890 -lt 549755813890 2024-03-29T06:48:14.794 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:48:14.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:14.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:15.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:48:15.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:15.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:15.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:15.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:15.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:15.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:15.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:15.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:48:15.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:15.486 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:15.486 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:48:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:48:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:48:15.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-29T06:48:15.862 INFO:tasks.workunit.client.0.smithi123.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-29T06:48:15.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:48:15.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-29T06:48:15.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-03-29T06:48:15.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:48:15.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:48:15.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:48:15.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-29T06:48:15.864 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-03-29T06:48:15.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:48:15.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:48:15.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:48:15.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-29T06:48:15.866 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-03-29T06:48:15.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:48:15.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:48:15.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:48:15.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-29T06:48:15.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-03-29T06:48:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:48:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:48:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:48:15.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-29T06:48:15.869 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:48:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:48:15.872 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:48:15.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:48:15.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:48:15.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:48:16.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:48:16.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-29T06:48:16.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:48:16.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:48:16.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:48:16.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:48:16.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:48:16.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-29T06:48:18.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:48:18.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:48:18.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:48:18.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:48:18.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:48:18.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:48:18.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:48:18.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:48:18.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:48:18.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:48:18.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:48:18.714 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:48:18.714 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:48:18.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:48:18.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:48:18.716 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:48:18.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:48:18.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:48:18.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:48:18.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:48:18.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:48:18.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:48:18.750 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:18.747+0000 7f238098b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:18.753 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:18.751+0000 7f238098b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:18.754 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:18.751+0000 7f238098b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:19.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:48:19.306 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:19.303+0000 7f238098b7c0 -1 Falling back to public interface 2024-03-29T06:48:19.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:20.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:20.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:20.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:48:20.424 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:48:20.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:20.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:48:20.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:21.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:21.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:21.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:48:21.732 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:48:21.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:21.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:48:21.969 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:21.967+0000 7f238098b7c0 -1 osd.0 130 log_to_monitors true 2024-03-29T06:48:22.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:23.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:23.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:23.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:48:23.034 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:48:23.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:23.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 134 up_thru 129 down_at 131 last_clean_interval [106,130) [v2:127.0.0.1:6802/1700300422,v1:127.0.0.1:6803/1700300422] [v2:127.0.0.1:6804/1700300422,v1:127.0.0.1:6805/1700300422] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:48:23.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:48:23.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:48:23.450 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:48:23.450 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:48:23.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:48:23.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:48:23.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:48:23.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:48:23.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:48:23.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:48:23.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:48:23.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:48:23.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:48:23.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:48:23.990 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:48:23.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:48:23.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:23.991 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:48:24.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=575525617666 2024-03-29T06:48:24.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 575525617666 2024-03-29T06:48:24.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617666' 2024-03-29T06:48:24.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:24.225 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:48:24.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=549755813893 2024-03-29T06:48:24.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 549755813893 2024-03-29T06:48:24.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617666 1-549755813893' 2024-03-29T06:48:24.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:24.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-575525617666 2024-03-29T06:48:24.472 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:24.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:48:24.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-575525617666 2024-03-29T06:48:24.474 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:24.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=575525617666 2024-03-29T06:48:24.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 575525617666' 2024-03-29T06:48:24.476 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 575525617666 2024-03-29T06:48:24.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:25.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617666 -lt 575525617666 2024-03-29T06:48:25.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:25.017 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-549755813893 2024-03-29T06:48:25.017 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:25.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:48:25.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:25.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-549755813893 2024-03-29T06:48:25.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=549755813893 2024-03-29T06:48:25.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 549755813893' 2024-03-29T06:48:25.021 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 549755813893 2024-03-29T06:48:25.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:48:25.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 549755813892 -lt 549755813893 2024-03-29T06:48:25.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:48:26.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:48:26.468 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:48:26.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 549755813893 -lt 549755813893 2024-03-29T06:48:26.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:48:26.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:26.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:27.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:48:27.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:27.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:27.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:27.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:27.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:27.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:27.162 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:27.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T06:48:27.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:27.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:27.500 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:27.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-03-29T06:48:27.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-29T06:48:27.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:48:27.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-29T06:48:27.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:48:27.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:48:27.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:48:27.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:27.965 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:27.965 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:27.965 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:27.965 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:27.966 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:27.966 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:28.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-29T06:48:28.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:28.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:28.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:28.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:48:28.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:48:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:48:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:48:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:48:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:48:29.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:48:29.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:48:29.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:29.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:29.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:29.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:29.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:29.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:29.256 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:48:29.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:29.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:29.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:48:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:48:29.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:48:31.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:48:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-29T06:48:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:48:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:48:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:48:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:48:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:48:31.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-03-29T06:48:33.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:48:33.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:48:33.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:48:33.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:48:33.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:48:33.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:48:33.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:48:33.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:48:33.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:48:33.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:48:33.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:48:33.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:48:33.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:48:33.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:48:33.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:48:33.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:48:33.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:48:33.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:48:33.179 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:48:33.179 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:48:33.179 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:48:33.179 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:33.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:33.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:48:33.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:48:33.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:48:33.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:48:33.181 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:48:33.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:48:33.181 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:48:33.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:48:33.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:48:33.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:48:33.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:48:33.209 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:33.206+0000 7f51e66e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:33.211 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:33.206+0000 7f51e66e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:33.213 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:33.210+0000 7f51e66e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:33.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:33.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:33.754 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:33.750+0000 7f51e66e27c0 -1 Falling back to public interface 2024-03-29T06:48:33.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:34.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:34.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:34.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:48:34.851 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:48:34.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:34.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:35.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:36.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:36.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:36.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:48:36.166 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:48:36.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:36.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:36.486 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:36.615 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:36.610+0000 7f51e66e27c0 -1 osd.1 136 log_to_monitors true 2024-03-29T06:48:37.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:37.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:37.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:48:37.489 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:48:37.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:37.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:37.962 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 140 up_thru 134 down_at 137 last_clean_interval [128,136) [v2:127.0.0.1:6810/108013050,v1:127.0.0.1:6811/108013050] [v2:127.0.0.1:6812/108013050,v1:127.0.0.1:6813/108013050] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:48:37.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:48:37.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:48:37.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:48:37.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:48:37.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:48:37.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:48:37.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:48:37.963 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:48:37.964 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:48:37.964 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:48:37.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:48:37.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:48:37.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:48:38.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:48:38.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:48:38.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:48:38.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:48:38.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:48:38.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:48:38.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:48:38.540 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:48:38.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:48:38.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:38.541 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:48:38.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=575525617670 2024-03-29T06:48:38.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 575525617670 2024-03-29T06:48:38.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617670' 2024-03-29T06:48:38.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:38.728 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:48:38.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=601295421442 2024-03-29T06:48:38.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 601295421442 2024-03-29T06:48:38.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617670 1-601295421442' 2024-03-29T06:48:38.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:38.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-575525617670 2024-03-29T06:48:38.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:38.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:48:38.946 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-575525617670 2024-03-29T06:48:38.947 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:38.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=575525617670 2024-03-29T06:48:38.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 575525617670' 2024-03-29T06:48:38.947 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 575525617670 2024-03-29T06:48:38.948 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:39.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617669 -lt 575525617670 2024-03-29T06:48:39.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:48:40.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:48:40.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:40.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617670 -lt 575525617670 2024-03-29T06:48:40.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:40.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:40.603 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-601295421442 2024-03-29T06:48:40.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:48:40.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-601295421442 2024-03-29T06:48:40.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:40.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=601295421442 2024-03-29T06:48:40.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 601295421442' 2024-03-29T06:48:40.606 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 601295421442 2024-03-29T06:48:40.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:48:40.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 601295421442 -lt 601295421442 2024-03-29T06:48:40.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:48:40.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:40.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:41.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:48:41.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:41.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:41.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:41.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:41.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:41.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:41.347 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:41.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:48:41.703 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:41.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:41.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:48:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:48:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:48:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:48:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:48:42.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:48:42.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:48:42.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:42.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:42.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:42.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:42.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:42.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:42.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:42.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:48:42.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:42.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:42.448 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:48:42.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:48:42.829 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:48:43.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:48:43.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:48:43.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:48:43.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:48:43.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:48:43.380 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:48:43.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:43.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:43.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:43.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:43.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:43.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:43.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:43.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:48:43.694 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:43.694 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:43.694 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:44.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:48:44.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:48:44.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:48:44.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:48:44.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:48:44.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:48:44.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:48:44.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:48:44.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:48:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:48:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:48:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:48:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:48:44.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:48:44.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:48:44.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:44.835 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:44.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:44.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:44.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:44.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:44.836 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:45.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:48:45.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:45.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:45.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:45.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:48:45.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:48:45.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:48:45.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-03-29T06:48:45.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:48:45.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-29T06:48:45.496 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-03-29T06:48:45.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:48:45.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:48:45.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:48:45.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:48:45.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:48:45.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-03-29T06:48:47.696 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-29T06:48:48.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:48:48.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:48:48.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:48:48.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:48:48.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:48:48.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:48:48.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:48:48.232 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:48:48.232 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:48:48.232 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:48:48.232 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:48:48.232 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:48:48.232 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:48:48.232 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:48:48.234 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:48:48.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:48:48.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:48:48.235 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:48:48.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:48:48.235 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:48:48.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:48:48.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:48:48.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:48:48.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:48:48.269 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:48.270+0000 7f621b24e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:48.271 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:48.270+0000 7f621b24e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:48.273 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:48.274+0000 7f621b24e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:48:48.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:48:48.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:48:48.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:48:48.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:48:48.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:48:48.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:48.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:48:48.667 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:48:48.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:48.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:48.817 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:48.818+0000 7f621b24e7c0 -1 Falling back to public interface 2024-03-29T06:48:49.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:50.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:50.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:50.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:48:50.027 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:48:50.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:50.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:50.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:51.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:51.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:51.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:48:51.329 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:48:51.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:51.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:51.581 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:48:51.582+0000 7f621b24e7c0 -1 osd.1 142 log_to_monitors true 2024-03-29T06:48:51.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:48:52.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:48:52.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:48:52.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:48:52.640 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:48:52.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:48:52.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:48:53.235 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 146 up_thru 146 down_at 143 last_clean_interval [140,142) [v2:127.0.0.1:6810/1212114488,v1:127.0.0.1:6811/1212114488] [v2:127.0.0.1:6812/1212114488,v1:127.0.0.1:6813/1212114488] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:48:53.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:48:53.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:48:53.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:48:53.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:48:53.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:48:53.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:48:53.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:48:53.236 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:48:53.236 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:48:53.236 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:48:53.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:48:53.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:48:53.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:48:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:48:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:48:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:48:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:48:53.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:48:53.359 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:48:53.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:48:53.676 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:48:53.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:48:53.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:53.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:48:53.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=575525617674 2024-03-29T06:48:53.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 575525617674 2024-03-29T06:48:53.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617674' 2024-03-29T06:48:53.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:48:53.824 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:48:54.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225218 2024-03-29T06:48:54.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225218 2024-03-29T06:48:54.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617674 1-627065225218' 2024-03-29T06:48:54.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:54.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-575525617674 2024-03-29T06:48:54.041 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:54.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:48:54.043 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-575525617674 2024-03-29T06:48:54.044 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:54.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=575525617674 2024-03-29T06:48:54.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 575525617674' 2024-03-29T06:48:54.045 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 575525617674 2024-03-29T06:48:54.045 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:54.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617672 -lt 575525617674 2024-03-29T06:48:54.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:48:55.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:48:55.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:48:55.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617674 -lt 575525617674 2024-03-29T06:48:55.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:48:55.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225218 2024-03-29T06:48:55.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:48:55.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:48:55.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225218 2024-03-29T06:48:55.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:48:55.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225218 2024-03-29T06:48:55.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225218' 2024-03-29T06:48:55.674 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 627065225218 2024-03-29T06:48:55.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:48:55.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225218 -lt 627065225218 2024-03-29T06:48:55.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:48:55.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:55.997 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:56.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:48:56.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:48:56.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:48:56.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:48:56.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:48:56.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:48:56.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:48:56.404 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:48:56.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:48:56.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:48:56.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:48:56.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:48:57.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:48:57.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:48:57.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:48:57.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:48:57.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-29T06:48:57.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-03-29T06:48:57.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:48:57.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:48:57.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-29T06:48:57.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:48:57.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:48:57.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:48:57.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:48:57.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-29T06:48:57.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:48:57.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:48:57.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:48:57.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:48:57.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:48:57.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-03-29T06:48:59.580 INFO:tasks.workunit.client.0.smithi123.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:49:00.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:49:00.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:49:00.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:49:00.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:49:00.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:49:00.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:49:00.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:00.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:49:00.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:49:00.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:49:00.375 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:49:00.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:49:00.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:49:00.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:49:00.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:49:00.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:49:00.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:49:00.410 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:00.410+0000 7fc28df0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:00.414 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:00.414+0000 7fc28df0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:00.414 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:00.414+0000 7fc28df0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:00.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:49:00.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:49:00.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:49:00.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:49:00.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:49:00.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:00.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:49:00.694 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:49:00.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:00.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:00.943 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:00.942+0000 7fc28df0b7c0 -1 Falling back to public interface 2024-03-29T06:49:01.061 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:02.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:02.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:02.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:49:02.062 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:49:02.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:02.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:02.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:03.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:03.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:03.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:49:03.410 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:49:03.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:03.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:03.713 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:03.714+0000 7fc28df0b7c0 -1 osd.0 148 log_to_monitors true 2024-03-29T06:49:03.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:04.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:04.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:04.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:49:04.728 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:49:04.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:04.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:05.034 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 152 up_thru 152 down_at 149 last_clean_interval [134,148) [v2:127.0.0.1:6802/157295845,v1:127.0.0.1:6803/157295845] [v2:127.0.0.1:6804/157295845,v1:127.0.0.1:6805/157295845] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:49:05.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:49:05.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:49:05.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:49:05.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:49:05.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:49:05.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:49:05.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:49:05.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:49:05.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:49:05.036 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:49:05.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:49:05.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:49:05.036 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:49:05.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:49:05.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:49:05.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:49:05.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:49:05.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:49:05.140 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:49:05.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:49:05.714 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:49:05.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:49:05.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:05.714 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:49:05.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028994 2024-03-29T06:49:05.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028994 2024-03-29T06:49:05.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994' 2024-03-29T06:49:05.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:05.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:49:06.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225221 2024-03-29T06:49:06.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225221 2024-03-29T06:49:06.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994 1-627065225221' 2024-03-29T06:49:06.279 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:06.280 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028994 2024-03-29T06:49:06.280 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:06.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:49:06.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028994 2024-03-29T06:49:06.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:06.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028994 2024-03-29T06:49:06.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028994' 2024-03-29T06:49:06.284 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 652835028994 2024-03-29T06:49:06.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:06.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028994 -lt 652835028994 2024-03-29T06:49:06.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:06.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225221 2024-03-29T06:49:06.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:06.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:49:06.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225221 2024-03-29T06:49:06.704 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:06.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225221 2024-03-29T06:49:06.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225221' 2024-03-29T06:49:06.706 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 627065225221 2024-03-29T06:49:06.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:49:07.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225221 -lt 627065225221 2024-03-29T06:49:07.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:49:07.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:07.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:07.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:49:07.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:07.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:07.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:07.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:07.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:07.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:07.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:49:07.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:07.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:07.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:08.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-29T06:49:08.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-29T06:49:08.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:49:08.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-29T06:49:08.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:49:08.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:49:08.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:49:08.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:08.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:08.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:08.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:08.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:08.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:08.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:08.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:49:08.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:08.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:08.509 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-29T06:49:08.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:49:08.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:49:08.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:49:09.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-29T06:49:11.269 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-29T06:49:11.269 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:49:11.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:49:11.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:49:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:49:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:49:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:49:11.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:11.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:49:11.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:49:11.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:49:11.817 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:49:11.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:49:11.817 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:49:11.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:49:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:49:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:49:11.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:49:11.850 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:11.849+0000 7f6ee48d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:11.853 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:11.853+0000 7f6ee48d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:11.854 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:11.853+0000 7f6ee48d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:12.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:49:12.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:49:12.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:49:12.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:49:12.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:49:12.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:12.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:49:12.135 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:49:12.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:12.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:12.398 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:12.397+0000 7f6ee48d37c0 -1 Falling back to public interface 2024-03-29T06:49:12.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:13.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:13.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:13.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:49:13.447 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:49:13.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:13.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:13.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:14.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:14.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:14.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:49:14.766 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:49:14.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:14.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:15.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:15.469 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:15.469+0000 7f6ee48d37c0 -1 osd.1 154 log_to_monitors true 2024-03-29T06:49:16.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:16.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:16.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:49:16.116 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:49:16.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:16.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:16.473 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 158 up_thru 153 down_at 155 last_clean_interval [146,154) [v2:127.0.0.1:6810/3239182424,v1:127.0.0.1:6811/3239182424] [v2:127.0.0.1:6812/3239182424,v1:127.0.0.1:6813/3239182424] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:49:16.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:49:16.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:49:16.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:49:16.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:49:16.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:49:16.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:49:16.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:49:16.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:49:16.475 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:49:16.475 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:49:16.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:49:16.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:49:16.475 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:49:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:49:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:49:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:49:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:49:16.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:49:16.579 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:49:17.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:49:17.062 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:49:17.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:49:17.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:17.063 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:49:17.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028997 2024-03-29T06:49:17.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028997 2024-03-29T06:49:17.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028997' 2024-03-29T06:49:17.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:17.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:49:17.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832770 2024-03-29T06:49:17.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832770 2024-03-29T06:49:17.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028997 1-678604832770' 2024-03-29T06:49:17.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:17.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028997 2024-03-29T06:49:17.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:17.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:49:17.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028997 2024-03-29T06:49:17.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:17.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028997 2024-03-29T06:49:17.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028997' 2024-03-29T06:49:17.550 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 652835028997 2024-03-29T06:49:17.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:17.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028996 -lt 652835028997 2024-03-29T06:49:17.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:49:18.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:49:18.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:19.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028997 -lt 652835028997 2024-03-29T06:49:19.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:19.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832770 2024-03-29T06:49:19.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:19.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:49:19.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832770 2024-03-29T06:49:19.196 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:19.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832770 2024-03-29T06:49:19.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832770' 2024-03-29T06:49:19.198 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 678604832770 2024-03-29T06:49:19.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:49:19.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832770 -lt 678604832770 2024-03-29T06:49:19.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:49:19.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:19.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:19.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:49:19.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:19.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:19.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:19.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:19.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:19.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:19.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:20.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:49:20.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:20.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:20.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:20.551 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:49:20.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:49:20.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:49:20.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:49:20.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:49:20.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:49:20.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:49:20.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:20.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:20.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:20.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:20.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:20.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:20.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:20.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:49:20.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:20.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:20.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:21.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:49:21.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:49:21.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:49:21.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:49:21.311 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-29T06:49:21.312 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:49:21.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:49:21.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:49:21.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:49:21.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:49:21.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:49:21.316 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:49:21.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:49:21.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:49:21.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:49:21.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:49:21.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:21.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:49:21.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:49:21.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:49:21.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:49:21.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:49:21.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:23.590 INFO:tasks.workunit.client.0.smithi123.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-29T06:49:24.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:49:24.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:49:24.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:49:24.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:49:24.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:49:24.513 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:49:24.514 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:49:24.514 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:49:24.514 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:49:24.514 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:49:24.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:49:24.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:49:24.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:49:24.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:49:24.516 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:49:24.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:49:24.517 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:49:24.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:49:24.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:49:24.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:49:24.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:49:24.550 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:24.549+0000 7f1f3655c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:24.553 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:24.553+0000 7f1f3655c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:24.554 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:24.553+0000 7f1f3655c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:24.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:25.090 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:25.089+0000 7f1f3655c7c0 -1 Falling back to public interface 2024-03-29T06:49:25.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:26.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:26.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:26.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:49:26.148 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:49:26.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:26.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:26.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:49:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:49:27.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:27.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:27.807 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:28.059 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:28.057+0000 7f1f3655c7c0 -1 osd.0 160 log_to_monitors true 2024-03-29T06:49:28.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:28.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:28.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:49:28.809 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:49:28.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:28.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:29.552 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 164 up_thru 159 down_at 161 last_clean_interval [152,160) [v2:127.0.0.1:6802/3281236956,v1:127.0.0.1:6803/3281236956] [v2:127.0.0.1:6804/3281236956,v1:127.0.0.1:6805/3281236956] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:49:29.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:49:29.554 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:49:29.554 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:49:29.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:49:29.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:49:29.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:49:29.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:49:29.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:49:29.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:49:29.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:49:29.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:49:29.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:49:30.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:49:30.201 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:49:30.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:49:30.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:30.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:49:30.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636546 2024-03-29T06:49:30.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636546 2024-03-29T06:49:30.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546' 2024-03-29T06:49:30.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:30.470 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:49:30.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832773 2024-03-29T06:49:30.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832773 2024-03-29T06:49:30.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546 1-678604832773' 2024-03-29T06:49:30.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:30.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636546 2024-03-29T06:49:30.838 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:30.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:49:30.839 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636546 2024-03-29T06:49:30.840 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:30.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636546 2024-03-29T06:49:30.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636546' 2024-03-29T06:49:30.841 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 704374636546 2024-03-29T06:49:30.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:31.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 704374636546 2024-03-29T06:49:31.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:49:32.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:49:32.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:32.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636546 -lt 704374636546 2024-03-29T06:49:32.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:32.576 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832773 2024-03-29T06:49:32.576 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:32.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:49:32.580 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832773 2024-03-29T06:49:32.580 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:32.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832773 2024-03-29T06:49:32.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832773' 2024-03-29T06:49:32.581 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 678604832773 2024-03-29T06:49:32.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:49:32.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832774 -lt 678604832773 2024-03-29T06:49:32.901 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:49:32.902 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:32.902 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:33.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:49:33.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:33.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:33.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:33.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:33.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:33.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:33.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:33.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:49:33.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:33.601 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:33.601 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:33.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:49:33.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:49:33.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:49:33.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:49:33.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:49:33.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:49:34.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:49:34.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:34.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:34.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:34.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:34.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:34.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:34.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:34.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:49:34.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:34.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:34.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:34.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:49:34.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:49:34.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:49:34.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:34.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:49:34.692 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:49:34.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:49:34.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:49:34.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:49:36.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:49:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:49:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:49:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:49:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:49:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:49:36.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-29T06:49:36.766 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-29T06:49:36.766 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-29T06:49:37.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:49:37.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:49:37.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:49:37.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:49:37.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:49:37.334 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:49:37.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:49:37.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:49:37.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:49:37.336 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:49:37.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:49:37.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:49:37.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:49:37.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:49:37.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:49:37.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:49:37.370 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:37.369+0000 7f7b8a3a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:37.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:37.373+0000 7f7b8a3a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:37.374 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:37.373+0000 7f7b8a3a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:37.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:49:37.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:37.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:37.922 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:37.921+0000 7f7b8a3a17c0 -1 Falling back to public interface 2024-03-29T06:49:38.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:39.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:39.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:39.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:49:39.005 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:49:39.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:39.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:39.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:40.274 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:40.273+0000 7f7b8a3a17c0 -1 osd.1 166 log_to_monitors true 2024-03-29T06:49:40.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:40.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:40.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:49:40.315 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:49:40.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:40.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:40.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:41.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:41.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:41.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:49:41.735 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:49:41.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:41.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:49:42.195 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 170 up_thru 170 down_at 167 last_clean_interval [158,166) [v2:127.0.0.1:6810/3415362329,v1:127.0.0.1:6811/3415362329] [v2:127.0.0.1:6812/3415362329,v1:127.0.0.1:6813/3415362329] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:49:42.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:49:42.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:49:42.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:49:42.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:49:42.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:49:42.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:49:42.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:49:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:49:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:49:42.197 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:49:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:49:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:49:42.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:49:42.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:49:42.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:49:42.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:49:42.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:49:42.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:49:42.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:49:42.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:49:42.781 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:49:42.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:49:42.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:42.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:49:42.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636549 2024-03-29T06:49:42.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636549 2024-03-29T06:49:42.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636549' 2024-03-29T06:49:42.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:42.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:49:43.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=730144440322 2024-03-29T06:49:43.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 730144440322 2024-03-29T06:49:43.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636549 1-730144440322' 2024-03-29T06:49:43.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:43.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636549 2024-03-29T06:49:43.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:43.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:49:43.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636549 2024-03-29T06:49:43.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:43.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636549 2024-03-29T06:49:43.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636549' 2024-03-29T06:49:43.150 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 704374636549 2024-03-29T06:49:43.150 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:43.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636548 -lt 704374636549 2024-03-29T06:49:43.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:49:44.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:49:44.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:44.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636550 -lt 704374636549 2024-03-29T06:49:44.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:44.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-730144440322 2024-03-29T06:49:44.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:44.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:49:44.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-730144440322 2024-03-29T06:49:44.761 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:44.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=730144440322 2024-03-29T06:49:44.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 730144440322' 2024-03-29T06:49:44.763 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 730144440322 2024-03-29T06:49:44.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:49:45.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 730144440322 -lt 730144440322 2024-03-29T06:49:45.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:49:45.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:45.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:45.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:49:45.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:45.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:45.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:45.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:45.476 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:45.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:45.477 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:45.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:49:45.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:45.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:45.772 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:46.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:49:46.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:49:46.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:49:46.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:49:46.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-29T06:49:46.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:49:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:49:46.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:49:46.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:49:46.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:46.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:49:46.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:49:46.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:49:46.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:49:46.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:49:46.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:48.254 INFO:tasks.workunit.client.0.smithi123.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-29T06:49:49.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:49:49.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:49:49.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:49:49.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:49:49.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:49:49.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:49:49.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:49:49.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:49:49.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:49:49.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:49:49.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:49:49.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:49:49.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:49:49.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:49:49.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:49:49.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:49:49.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:49:49.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:49:49.214 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:49:49.214 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:49:49.214 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:49:49.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:49.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:49:49.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:49:49.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:49:49.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:49:49.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:49:49.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:49:49.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:49:49.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:49:49.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:49:49.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:49:49.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:49:49.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:49:49.216 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:49:49.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:49:49.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:49:49.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:49:49.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:49:49.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:49:49.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:49:49.244 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:49.245+0000 7fb487c827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:49.246 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:49.245+0000 7fb487c827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:49.248 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:49.249+0000 7fb487c827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:49:49.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:49:49.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:49.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:49.781 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:49.781+0000 7fb487c827c0 -1 Falling back to public interface 2024-03-29T06:49:49.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:50.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:50.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:50.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:49:50.859 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:49:50.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:50.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:51.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:52.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:52.167 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:52.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:49:52.168 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:49:52.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:52.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:52.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:49:52.688 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:49:52.689+0000 7fb487c827c0 -1 osd.0 172 log_to_monitors true 2024-03-29T06:49:53.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:49:53.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:49:53.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:49:53.473 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:49:53.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:49:53.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:49:53.805 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 176 up_thru 170 down_at 173 last_clean_interval [164,172) [v2:127.0.0.1:6802/661055712,v1:127.0.0.1:6803/661055712] [v2:127.0.0.1:6804/661055712,v1:127.0.0.1:6805/661055712] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:49:53.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:49:53.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:49:53.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:49:53.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:49:53.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:49:53.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:49:53.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:49:53.806 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:49:53.806 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:49:53.806 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:49:53.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:49:53.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:49:53.807 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:49:53.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:49:53.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:49:53.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:49:53.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:49:53.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:49:53.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:49:54.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:49:54.285 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:49:54.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:49:54.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:54.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:49:54.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=755914244098 2024-03-29T06:49:54.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 755914244098 2024-03-29T06:49:54.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-755914244098' 2024-03-29T06:49:54.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:49:54.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:49:54.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=730144440325 2024-03-29T06:49:54.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 730144440325 2024-03-29T06:49:54.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-755914244098 1-730144440325' 2024-03-29T06:49:54.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:54.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:54.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-755914244098 2024-03-29T06:49:54.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:49:54.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-755914244098 2024-03-29T06:49:54.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=755914244098 2024-03-29T06:49:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 755914244098' 2024-03-29T06:49:54.884 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 755914244098 2024-03-29T06:49:54.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:55.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 755914244098 2024-03-29T06:49:55.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:49:56.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:49:56.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:49:56.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 755914244098 -lt 755914244098 2024-03-29T06:49:56.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:49:56.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:49:56.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-730144440325 2024-03-29T06:49:56.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:49:56.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-730144440325 2024-03-29T06:49:56.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:49:56.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=730144440325 2024-03-29T06:49:56.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 730144440325' 2024-03-29T06:49:56.610 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 730144440325 2024-03-29T06:49:56.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:49:56.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 730144440326 -lt 730144440325 2024-03-29T06:49:56.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:49:56.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:56.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:57.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:49:57.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:49:57.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:49:57.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:49:57.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:49:57.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:49:57.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:49:57.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:49:57.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:49:57.608 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:49:57.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:49:57.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:49:57.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:49:57.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:49:57.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:49:57.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:49:57.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:49:57.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:49:57.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:49:57.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:49:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-29T06:49:59.652 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-29T06:49:59.653 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-29T06:50:00.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:50:00.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:50:00.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:50:00.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:50:00.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:50:00.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:50:00.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:50:00.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:50:00.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:50:00.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:50:00.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:50:00.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:50:00.223 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:50:00.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:50:00.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:50:00.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:50:00.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:50:00.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:50:00.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:50:00.258 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:00.256+0000 7f43e7b057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:00.261 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:00.260+0000 7f43e7b057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:00.262 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:00.260+0000 7f43e7b057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:00.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:50:00.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:50:00.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:50:00.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:50:00.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:50:00.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:00.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:50:00.535 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:50:00.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:00.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:00.809 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:00.808+0000 7f43e7b057c0 -1 Falling back to public interface 2024-03-29T06:50:00.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:01.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:01.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:01.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:50:01.842 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:50:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:01.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:02.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:02.998 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:02.996+0000 7f43e7b057c0 -1 osd.1 178 log_to_monitors true 2024-03-29T06:50:03.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:03.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:03.164 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:50:03.164 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:50:03.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:03.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:03.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:04.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:04.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:04.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:50:04.563 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:50:04.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:04.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 182 up_thru 182 down_at 179 last_clean_interval [170,178) [v2:127.0.0.1:6810/3028652661,v1:127.0.0.1:6811/3028652661] [v2:127.0.0.1:6812/3028652661,v1:127.0.0.1:6813/3028652661] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:50:05.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:50:05.049 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:50:05.049 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:50:05.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:50:05.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:50:05.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:50:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:50:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:50:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:50:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:50:05.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:50:05.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:50:05.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:50:05.573 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:50:05.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:50:05.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:05.574 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:50:05.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=755914244101 2024-03-29T06:50:05.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 755914244101 2024-03-29T06:50:05.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-755914244101' 2024-03-29T06:50:05.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:05.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:50:05.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=781684047874 2024-03-29T06:50:05.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 781684047874 2024-03-29T06:50:05.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-755914244101 1-781684047874' 2024-03-29T06:50:05.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:05.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-755914244101 2024-03-29T06:50:05.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:05.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:50:05.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-755914244101 2024-03-29T06:50:05.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:05.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=755914244101 2024-03-29T06:50:05.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 755914244101' 2024-03-29T06:50:05.870 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 755914244101 2024-03-29T06:50:05.870 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:50:06.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 755914244100 -lt 755914244101 2024-03-29T06:50:06.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:50:07.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:50:07.171 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:50:07.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 755914244101 -lt 755914244101 2024-03-29T06:50:07.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:07.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-781684047874 2024-03-29T06:50:07.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:07.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:50:07.490 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-781684047874 2024-03-29T06:50:07.491 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:07.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=781684047874 2024-03-29T06:50:07.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 781684047874' 2024-03-29T06:50:07.492 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 781684047874 2024-03-29T06:50:07.492 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:50:07.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 781684047874 -lt 781684047874 2024-03-29T06:50:07.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:50:07.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:07.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:08.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:50:08.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:50:08.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:50:08.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:50:08.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:50:08.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:50:08.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:50:08.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:50:08.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:50:08.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:50:08.548 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:08.548 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:50:08.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:50:08.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:50:10.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:50:10.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-29T06:50:10.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:50:10.214 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:50:10.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:50:10.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:50:10.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:50:10.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-03-29T06:50:11.127 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-29T06:50:11.127 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-29T06:50:11.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:50:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:50:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:50:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:50:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:50:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:50:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:50:11.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:50:11.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:50:11.675 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:50:11.675 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:50:11.675 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:50:11.675 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:11.675 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:11.675 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:11.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:11.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:50:11.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:50:11.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:50:11.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:50:11.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:50:11.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:50:11.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:50:11.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:50:11.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:50:11.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:50:11.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:50:11.680 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:50:11.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:50:11.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:50:11.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:50:11.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:50:11.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:50:11.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:50:11.710 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:11.708+0000 7f98f37557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:11.712 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:11.712+0000 7f98f37557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:11.714 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:11.712+0000 7f98f37557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:11.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:12.257 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:12.256+0000 7f98f37557c0 -1 Falling back to public interface 2024-03-29T06:50:12.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:13.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:13.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:13.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:50:13.318 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:50:13.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:13.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:13.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:14.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:14.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:14.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:50:14.628 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:50:14.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:14.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:14.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:14.960 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:14.960+0000 7f98f37557c0 -1 osd.1 184 log_to_monitors true 2024-03-29T06:50:15.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:15.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:15.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:50:15.952 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:50:15.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:15.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 188 up_thru 188 down_at 185 last_clean_interval [182,184) [v2:127.0.0.1:6810/2449463920,v1:127.0.0.1:6811/2449463920] [v2:127.0.0.1:6812/2449463920,v1:127.0.0.1:6813/2449463920] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:50:16.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:50:16.314 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:50:16.314 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:50:16.314 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:50:16.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:50:16.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:50:16.315 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:50:16.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:50:16.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:50:16.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:50:16.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:50:16.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:50:16.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:50:16.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:50:16.965 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:50:16.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:50:16.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:16.966 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:50:17.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=755914244104 2024-03-29T06:50:17.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 755914244104 2024-03-29T06:50:17.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-755914244104' 2024-03-29T06:50:17.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:17.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:50:17.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=807453851650 2024-03-29T06:50:17.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 807453851650 2024-03-29T06:50:17.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-755914244104 1-807453851650' 2024-03-29T06:50:17.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:17.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-755914244104 2024-03-29T06:50:17.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:17.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:50:17.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-755914244104 2024-03-29T06:50:17.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:17.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=755914244104 2024-03-29T06:50:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 755914244104' 2024-03-29T06:50:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 755914244104 2024-03-29T06:50:17.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:50:17.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 755914244103 -lt 755914244104 2024-03-29T06:50:17.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:50:18.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:50:18.895 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:50:19.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 755914244105 -lt 755914244104 2024-03-29T06:50:19.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:19.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:19.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-807453851650 2024-03-29T06:50:19.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:50:19.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-807453851650 2024-03-29T06:50:19.220 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:19.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=807453851650 2024-03-29T06:50:19.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 807453851650' 2024-03-29T06:50:19.222 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 807453851650 2024-03-29T06:50:19.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:50:19.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 807453851650 -lt 807453851650 2024-03-29T06:50:19.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:50:19.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:19.554 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:19.990 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:50:19.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:50:19.991 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:50:19.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:50:19.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:50:19.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:50:19.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:50:19.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:50:20.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:50:20.318 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:50:20.318 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:20.318 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:20.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:50:20.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:50:20.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:50:20.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-29T06:50:20.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-29T06:50:20.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-03-29T06:50:20.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-29T06:50:20.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-29T06:50:20.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-29T06:50:20.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-29T06:50:20.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-29T06:50:20.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-29T06:50:20.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:50:21.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-29T06:50:21.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-03-29T06:50:21.177 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0: {} 2024-03-29T06:50:21.177 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-29T06:50:21.193 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1: {} 2024-03-29T06:50:21.193 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-29T06:50:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-29T06:50:21.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:50:21.209 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-29T06:50:21.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:50:21.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-29T06:50:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-29T06:50:21.210 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:50:21.511 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:50:21.511 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:50:21.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:50:21.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:50:21.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:50:21.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-29T06:50:21.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:50:21.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:50:21.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-29T06:50:21.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:50:21.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-29T06:50:21.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:50:21.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-29T06:50:21.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:50:21.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:50:21.515 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:50:21.515 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-29T06:50:21.515 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:50:21.515 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:50:21.515 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:21.515 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:21.516 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:21.516 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:50:21.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:50:21.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:50:21.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:50:21.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:50:21.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-29T06:50:21.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:50:21.622 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:50:21.623 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:21.623 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:21.623 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:21.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:50:21.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:50:21.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-29T06:50:21.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-29T06:50:21.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:50:21.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:50:21.737 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-29T06:50:21.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:50:21.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-29T06:50:21.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:50:21.739 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-29T06:50:21.739 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-29T06:50:21.739 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:50:22.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:50:22.027 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:50:22.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:50:22.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:50:22.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:50:22.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-29T06:50:22.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:50:22.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:50:22.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-29T06:50:22.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:50:22.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-29T06:50:22.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:50:22.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-29T06:50:22.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:50:22.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:50:22.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:50:22.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-29T06:50:22.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:50:22.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:50:22.030 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:22.031 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:22.031 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:22.031 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:50:22.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:50:22.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:50:22.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:50:22.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-29T06:50:22.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:50:22.147 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:50:22.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:50:22.148 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:22.148 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:22.148 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:22.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:50:22.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:50:22.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-29T06:50:22.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-29T06:50:22.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:50:22.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:50:22.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-29T06:50:22.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:50:22.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:50:22.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:50:22.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-29T06:50:22.269 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-29T06:50:22.269 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:50:22.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:50:22.592 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:50:22.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:50:22.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:50:22.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:50:22.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-29T06:50:22.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:50:22.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:50:22.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-29T06:50:22.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:50:22.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-29T06:50:22.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:50:22.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-29T06:50:22.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:50:22.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:50:22.595 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:50:22.595 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-29T06:50:22.596 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:50:22.596 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:50:22.596 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:22.596 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:22.596 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:22.596 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:50:22.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:50:22.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:50:22.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:50:22.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:50:22.712 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-29T06:50:22.712 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:50:22.712 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:50:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:22.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:50:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:50:22.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-29T06:50:22.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-29T06:50:22.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:50:22.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:50:22.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:50:22.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-29T06:50:22.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:50:22.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-29T06:50:22.826 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:50:22.826 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-29T06:50:23.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:50:23.128 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:50:23.128 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:50:23.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:50:23.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:50:23.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-29T06:50:23.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:50:23.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:50:23.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-29T06:50:23.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:50:23.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-29T06:50:23.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:50:23.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-29T06:50:23.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:50:23.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:50:23.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:50:23.131 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-29T06:50:23.131 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:50:23.131 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:50:23.131 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:23.131 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:23.131 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:23.131 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:50:23.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:50:23.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:50:23.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:50:23.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:50:23.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-29T06:50:23.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:50:23.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:50:23.240 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:50:23.241 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:23.241 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:23.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:50:23.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:50:23.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-29T06:50:23.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-03-29T06:50:23.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-29T06:50:23.348 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-29T06:50:23.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:50:23.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:50:23.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:50:23.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:50:23.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-29T06:45:20.457527+0000 2024-03-29T06:50:23.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-29T06:50:23.899 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-29T06:50:23.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-29T06:45:20.457527+0000 2024-03-29T06:50:23.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:50:23.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:45:20.457527+0000 2024-03-29T06:50:23.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:50:23.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:50:23.916 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:50:23.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:50:23.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:50:23.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:50:23.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:50:23.916 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:50:24.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:50:24.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:50:25.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:50:25.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:50:25.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:50:25.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:50:25.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:50:25.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:50:25.207 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:50:25.516 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:50:25.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:50:26.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:50:26.518 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:50:26.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:50:26.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:50:26.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:50:26.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:50:26.519 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:50:26.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:50:24.858262+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-29T06:50:26.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-29T06:50:26.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-03-29T06:50:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-03-29T06:50:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-03-29T06:50:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.891 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:26.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:26.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:27.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:27.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:27.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:27.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:27.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:27.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:27.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:50:27.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-03-29T06:50:27.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-29T06:50:27.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-29T06:50:27.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-29T06:50:27.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-29T06:50:27.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-29T06:50:27.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-29T06:50:27.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=188 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:50:27.207 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:50:27.208 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T06:50:27.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-29T06:50:27.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:50:27.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-29T06:50:27.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-29T06:50:27.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:50:27.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:50:27.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:50:27.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-03-29T06:50:27.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-29T06:50:27.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-29T06:50:27.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-03-29T06:50:27.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:50:27.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:50:27.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:50:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:50:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:50:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:50:27.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:50:27.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:50:27.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:50:27.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:50:27.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:50:27.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:50:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:50:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-29T06:50:29.547 INFO:tasks.workunit.client.0.smithi123.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-29T06:50:30.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:50:30.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:50:30.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:50:30.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:50:30.104 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:50:30.105 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:50:30.105 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:30.105 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:30.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:30.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:30.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:50:30.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:50:30.106 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:50:30.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:50:30.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:50:30.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:50:30.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:50:30.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:50:30.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:50:30.141 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:30.140+0000 7f9c88b307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:30.148 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:30.144+0000 7f9c88b307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:30.148 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:30.144+0000 7f9c88b307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:50:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:50:30.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:50:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:50:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:50:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:50:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:50:30.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:50:30.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:30.680 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:30.680+0000 7f9c88b307c0 -1 Falling back to public interface 2024-03-29T06:50:30.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:31.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:31.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:31.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:50:31.783 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:50:31.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:31.784 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:50:32.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:33.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:33.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:33.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:50:33.100 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:50:33.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:33.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:50:33.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:33.670 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:33.668+0000 7f9c88b307c0 -1 osd.0 190 log_to_monitors true 2024-03-29T06:50:34.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:34.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:34.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:50:34.406 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:50:34.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:34.407 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:50:34.881 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 194 up_thru 188 down_at 191 last_clean_interval [176,190) [v2:127.0.0.1:6802/1062222516,v1:127.0.0.1:6803/1062222516] [v2:127.0.0.1:6804/1062222516,v1:127.0.0.1:6805/1062222516] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:50:34.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:50:34.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:50:34.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:50:34.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:50:34.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:50:34.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:50:34.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:50:34.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:50:34.882 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:50:34.882 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:50:34.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:50:34.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:50:34.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:50:35.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:50:35.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:50:35.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:50:35.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:50:35.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:50:35.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:50:35.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:50:35.751 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:50:35.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:50:35.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:35.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:50:35.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=833223655426 2024-03-29T06:50:35.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 833223655426 2024-03-29T06:50:35.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-833223655426' 2024-03-29T06:50:35.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:35.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:50:36.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=807453851655 2024-03-29T06:50:36.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 807453851655 2024-03-29T06:50:36.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-833223655426 1-807453851655' 2024-03-29T06:50:36.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:36.113 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-833223655426 2024-03-29T06:50:36.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:36.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:50:36.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-833223655426 2024-03-29T06:50:36.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:36.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=833223655426 2024-03-29T06:50:36.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 833223655426' 2024-03-29T06:50:36.116 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 833223655426 2024-03-29T06:50:36.117 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:50:36.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 833223655426 -lt 833223655426 2024-03-29T06:50:36.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:36.507 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-807453851655 2024-03-29T06:50:36.507 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:36.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:50:36.510 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-807453851655 2024-03-29T06:50:36.510 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:36.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=807453851655 2024-03-29T06:50:36.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 807453851655' 2024-03-29T06:50:36.512 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 807453851655 2024-03-29T06:50:36.512 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:50:36.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 807453851655 -lt 807453851655 2024-03-29T06:50:36.841 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:50:36.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:36.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:50:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:50:37.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:50:37.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:50:37.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:50:37.237 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:50:37.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:50:37.238 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:50:37.580 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:50:37.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:50:37.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:37.581 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:37.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:50:37.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-29T06:50:37.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T06:50:37.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-29T06:50:37.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:50:37.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T06:50:38.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:50:38.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:50:38.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:50:38.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:50:38.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:50:38.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:50:38.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:50:38.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:50:38.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:50:38.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:50:38.375 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:38.376 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:50:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:50:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:50:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:50:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:50:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:50:38.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:50:38.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:50:38.735 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:50:39.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:50:39.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:50:39.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T06:50:39.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:50:39.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T06:50:39.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:50:39.287 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:50:39.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:50:39.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:50:39.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:50:39.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:50:39.288 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:50:39.288 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:50:39.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-29T06:50:39.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:50:39.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:39.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:39.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-29T06:50:39.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-29T06:50:39.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T06:50:39.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T06:50:39.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T06:50:39.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T06:50:39.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T06:50:39.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T06:50:39.974 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T06:50:40.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T06:50:40.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T06:50:40.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-29T06:50:40.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T06:50:40.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-29T06:50:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T06:50:40.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:50:40.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:50:40.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:50:40.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:50:40.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:50:40.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:50:40.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:50:41.073 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:50:41.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:50:41.074 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:41.075 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:41.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:50:41.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:50:41.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:50:41.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:50:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:50:41.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:50:41.426 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:50:41.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:50:41.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:50:41.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:50:41.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-03-29T06:50:43.378 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-29T06:50:43.378 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-29T06:50:43.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:50:43.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:43.940 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:50:43.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:50:43.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:50:43.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:50:43.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:50:43.943 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:50:43.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:50:43.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:50:43.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:50:43.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:50:43.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:50:43.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:50:43.981 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:43.979+0000 7ff72a4407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:43.984 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:43.983+0000 7ff72a4407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:43.985 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:43.983+0000 7ff72a4407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:50:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:50:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:50:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:50:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:50:44.254 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:44.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:50:44.255 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:50:44.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:44.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:44.540 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:44.539+0000 7ff72a4407c0 -1 Falling back to public interface 2024-03-29T06:50:44.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:45.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:45.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:45.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:50:45.642 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:50:45.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:45.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:45.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:46.884 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:46.883+0000 7ff72a4407c0 -1 osd.1 196 log_to_monitors true 2024-03-29T06:50:46.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:46.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:46.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:50:46.961 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:50:46.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:46.962 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:47.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:48.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:48.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:48.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:50:48.361 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:50:48.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:48.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:50:48.765 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 200 up_thru 200 down_at 197 last_clean_interval [188,196) [v2:127.0.0.1:6810/2176785501,v1:127.0.0.1:6811/2176785501] [v2:127.0.0.1:6812/2176785501,v1:127.0.0.1:6813/2176785501] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:50:48.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:50:48.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:50:48.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:50:48.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:50:48.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:50:48.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:50:48.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:50:48.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:50:48.767 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:50:48.767 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:50:48.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:50:48.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:50:48.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:50:48.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:50:48.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:50:48.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:50:48.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:50:48.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:50:48.896 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:50:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:50:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:50:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:50:49.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:49.278 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:50:49.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=833223655430 2024-03-29T06:50:49.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 833223655430 2024-03-29T06:50:49.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-833223655430' 2024-03-29T06:50:49.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:50:49.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:50:49.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459202 2024-03-29T06:50:49.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459202 2024-03-29T06:50:49.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-833223655430 1-858993459202' 2024-03-29T06:50:49.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:49.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-833223655430 2024-03-29T06:50:49.563 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:49.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:50:49.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:49.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-833223655430 2024-03-29T06:50:49.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=833223655430 2024-03-29T06:50:49.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 833223655430' 2024-03-29T06:50:49.566 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 833223655430 2024-03-29T06:50:49.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:50:49.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 833223655428 -lt 833223655430 2024-03-29T06:50:49.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:50:50.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:50:50.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:50:51.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 833223655430 -lt 833223655430 2024-03-29T06:50:51.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:50:51.177 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-858993459202 2024-03-29T06:50:51.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:50:51.179 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:50:51.180 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-858993459202 2024-03-29T06:50:51.181 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:50:51.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459202 2024-03-29T06:50:51.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 858993459202' 2024-03-29T06:50:51.182 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 858993459202 2024-03-29T06:50:51.182 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:50:51.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459202 -lt 858993459202 2024-03-29T06:50:51.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:50:51.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:51.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:51.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:50:51.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:50:51.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:50:51.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:50:51.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:50:51.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:50:51.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:50:51.945 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:50:52.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:50:52.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:50:52.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:50:52.285 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:50:52.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:50:52.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:50:52.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:50:52.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-03-29T06:50:52.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:50:52.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:50:52.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:50:52.856 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:50:52.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:50:52.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:50:52.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:50:53.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:50:55.090 INFO:tasks.workunit.client.0.smithi123.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:50:55.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:50:55.636 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:50:55.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:50:55.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:50:55.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:50:55.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:50:55.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:50:55.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:50:55.639 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:50:55.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:50:55.640 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:50:55.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:50:55.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:50:55.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:50:55.641 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:50:55.673 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:55.671+0000 7f3aaf3277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:55.677 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:55.675+0000 7f3aaf3277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:55.677 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:55.675+0000 7f3aaf3277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:50:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:50:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:50:55.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:50:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:50:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:50:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:50:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:50:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:55.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:50:56.216 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:56.215+0000 7f3aaf3277c0 -1 Falling back to public interface 2024-03-29T06:50:56.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:57.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:57.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:57.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:50:57.297 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:50:57.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:57.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:50:57.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:58.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:58.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:58.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:50:58.604 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:50:58.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:58.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:50:58.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:50:59.341 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:50:59.339+0000 7f3aaf3277c0 -1 osd.0 202 log_to_monitors true 2024-03-29T06:50:59.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:50:59.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:50:59.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:50:59.984 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:50:59.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:50:59.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 206 up_thru 200 down_at 203 last_clean_interval [194,202) [v2:127.0.0.1:6802/1487457437,v1:127.0.0.1:6803/1487457437] [v2:127.0.0.1:6804/1487457437,v1:127.0.0.1:6805/1487457437] exists,up 72c5bf1f-765a-483b-97a4-bdfd228a01f9 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:51:00.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:51:00.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:51:00.458 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:51:00.458 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:51:00.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:51:00.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:51:00.459 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:51:00.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:51:00.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:51:00.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:51:00.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:51:00.582 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:51:00.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:51:01.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:51:01.022 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:51:01.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:51:01.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:51:01.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:51:01.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=884763262978 2024-03-29T06:51:01.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 884763262978 2024-03-29T06:51:01.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-884763262978' 2024-03-29T06:51:01.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:51:01.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:51:01.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=858993459205 2024-03-29T06:51:01.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 858993459205 2024-03-29T06:51:01.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-884763262978 1-858993459205' 2024-03-29T06:51:01.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:51:01.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-884763262978 2024-03-29T06:51:01.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:51:01.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:51:01.602 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-884763262978 2024-03-29T06:51:01.602 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:51:01.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=884763262978 2024-03-29T06:51:01.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 884763262978' 2024-03-29T06:51:01.604 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 884763262978 2024-03-29T06:51:01.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:51:01.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 884763262978 2024-03-29T06:51:01.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:51:02.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:51:02.938 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:51:03.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262978 -lt 884763262978 2024-03-29T06:51:03.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:51:03.254 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-858993459205 2024-03-29T06:51:03.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:51:03.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:51:03.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-858993459205 2024-03-29T06:51:03.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:51:03.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=858993459205 2024-03-29T06:51:03.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 858993459205' 2024-03-29T06:51:03.259 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 858993459205 2024-03-29T06:51:03.259 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:51:03.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 858993459206 -lt 858993459205 2024-03-29T06:51:03.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:51:03.596 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:51:03.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:51:03.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:51:03.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:51:03.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:51:03.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:51:03.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:51:03.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:51:03.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:51:03.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:51:04.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:51:04.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:51:04.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:51:04.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:51:04.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:51:04.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:51:04.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:51:04.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:51:05.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:51:05.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:51:05.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:51:05.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:51:05.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:51:05.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:51:05.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:51:05.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-29T06:51:06.582 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-29T06:51:06.583 INFO:tasks.workunit.client.0.smithi123.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-29T06:51:07.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:51:07.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:51:07.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:51:07.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:51:07.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:51:07.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:51:07.118 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:51:07.119 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:51:07.119 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:07.119 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:07.119 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:51:07.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:51:07.120 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:51:07.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:51:07.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:51:07.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:51:07.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:51:07.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:51:07.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:51:07.159 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:07.159+0000 7fb9c8d577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:07.162 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:07.163+0000 7fb9c8d577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:07.163 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:07.163+0000 7fb9c8d577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:07.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:51:07.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:51:07.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:51:07.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:51:07.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:51:07.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:07.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:51:07.443 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:51:07.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:07.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:51:07.705 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:07.703+0000 7fb9c8d577c0 -1 Falling back to public interface 2024-03-29T06:51:07.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:08.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:08.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:08.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:51:08.793 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:51:08.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:08.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:51:09.172 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:10.046 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:10.047+0000 7fb9c8d577c0 -1 osd.1 208 log_to_monitors true 2024-03-29T06:51:10.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:10.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:10.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:51:10.174 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:51:10.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:10.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:51:10.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:11.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:11.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:11.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:51:11.564 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:51:11.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:11.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:51:11.877 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 212 up_thru 212 down_at 209 last_clean_interval [200,208) [v2:127.0.0.1:6810/1575070539,v1:127.0.0.1:6811/1575070539] [v2:127.0.0.1:6812/1575070539,v1:127.0.0.1:6813/1575070539] exists,up 4d3d9a88-c87a-446c-af5e-3e0717b128cf 2024-03-29T06:51:11.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:51:11.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:51:11.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:51:11.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:51:11.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:51:11.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:51:11.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:51:11.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:51:11.879 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:51:11.879 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:51:11.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:51:11.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:51:11.879 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:51:11.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:51:11.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:51:11.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:51:11.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:51:11.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:51:11.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:51:12.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:51:12.298 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:51:12.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:51:12.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:51:12.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:51:12.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=884763262981 2024-03-29T06:51:12.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 884763262981 2024-03-29T06:51:12.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-884763262981' 2024-03-29T06:51:12.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:51:12.529 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:51:12.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=910533066754 2024-03-29T06:51:12.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 910533066754 2024-03-29T06:51:12.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-884763262981 1-910533066754' 2024-03-29T06:51:12.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:51:12.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-884763262981 2024-03-29T06:51:12.672 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:51:12.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:51:12.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-884763262981 2024-03-29T06:51:12.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:51:12.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=884763262981 2024-03-29T06:51:12.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 884763262981' 2024-03-29T06:51:12.676 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 884763262981 2024-03-29T06:51:12.676 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:51:13.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262980 -lt 884763262981 2024-03-29T06:51:13.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:51:14.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:51:14.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:51:14.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262980 -lt 884763262981 2024-03-29T06:51:14.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:51:15.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T06:51:15.328 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:51:15.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262981 -lt 884763262981 2024-03-29T06:51:15.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:51:15.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-910533066754 2024-03-29T06:51:15.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:51:15.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:51:15.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-910533066754 2024-03-29T06:51:15.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:51:15.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=910533066754 2024-03-29T06:51:15.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 910533066754' 2024-03-29T06:51:15.643 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 910533066754 2024-03-29T06:51:15.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:51:15.968 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 910533066754 -lt 910533066754 2024-03-29T06:51:15.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:51:15.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:51:15.969 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:51:16.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:51:16.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:51:16.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:51:16.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:51:16.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:51:16.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:51:16.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:51:16.380 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:51:16.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:51:16.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:51:16.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:51:16.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:51:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:51:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:51:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:51:17.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:51:17.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:51:17.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-29T06:51:17.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:51:17.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-29T06:51:17.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:51:17.035 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-29T06:51:17.035 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:51:17.035 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:51:17.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:51:17.351 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-29T06:51:17.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:51:17.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-29T06:51:17.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:51:17.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-29T06:51:17.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:51:17.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:51:17.352 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:51:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-29T06:51:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:51:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:51:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:17.353 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:17.354 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:51:17.354 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:51:17.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:51:17.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:51:17.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:51:17.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:51:17.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-29T06:51:17.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:51:17.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:51:17.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:51:17.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-29T06:51:17.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:51:17.589 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-29T06:51:17.589 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:51:17.589 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-29T06:51:17.589 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-29T06:51:17.589 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:51:17.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:51:17.882 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:51:17.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:51:17.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:51:17.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:51:17.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-29T06:51:17.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:51:17.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:51:17.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-29T06:51:17.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:51:17.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-29T06:51:17.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:51:17.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-29T06:51:17.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:51:17.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:51:17.886 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:51:17.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-29T06:51:17.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:51:17.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:51:17.886 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:17.886 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:17.887 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:17.887 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:51:17.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:51:17.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:51:17.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:51:17.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:51:17.994 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-29T06:51:17.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:51:17.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:51:17.996 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:17.996 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:17.996 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:17.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:51:17.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:51:17.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-29T06:51:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-29T06:51:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:51:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:51:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-29T06:51:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:51:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:51:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:51:18.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-29T06:51:18.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:51:18.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:51:18.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:51:18.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-29T06:51:18.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:51:18.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-29T06:51:18.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:51:18.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-29T06:51:18.115 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-29T06:51:18.115 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:51:18.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:51:18.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-29T06:51:18.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:51:18.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-29T06:51:18.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:51:18.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-29T06:51:18.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:51:18.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:51:18.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:51:18.440 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-29T06:51:18.440 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:51:18.440 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:51:18.441 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:18.441 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:18.441 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:18.441 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:51:18.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:51:18.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:51:18.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:51:18.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:51:18.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-29T06:51:18.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-29T06:51:18.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-29T06:51:18.552 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:18.552 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:18.552 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:18.552 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.0.asok 2024-03-29T06:51:18.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:51:18.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-29T06:51:18.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-29T06:51:18.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-29T06:51:18.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-29T06:51:18.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-29T06:51:18.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-29T06:51:18.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-29T06:51:18.663 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-29T06:51:18.663 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-29T06:51:19.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-29T06:51:19.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-29T06:51:19.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-29T06:51:19.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-29T06:51:19.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-29T06:51:19.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-29T06:51:19.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-29T06:51:19.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-29T06:51:19.025 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-29T06:51:19.025 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-29T06:51:19.025 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:51:19.025 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:51:19.026 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:19.026 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:19.026 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:19.026 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:51:19.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61544/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-29T06:51:19.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-29T06:51:19.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-29T06:51:19.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-29T06:51:19.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-29T06:51:19.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-29T06:51:19.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-29T06:51:19.138 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:19.138 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:19.139 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:19.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-osd.1.asok 2024-03-29T06:51:19.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-29T06:51:19.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.61544/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-29T06:51:19.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-03-29T06:51:19.389 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0: {} 2024-03-29T06:51:19.389 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0: osd_max_object_size = '' 2024-03-29T06:51:19.404 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1: {} 2024-03-29T06:51:19.405 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1: osd_max_object_size = '' 2024-03-29T06:51:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-29T06:51:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-29T06:51:19.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:51:19.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:19.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:51:19.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:19.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:51:19.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:19.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-29T06:51:20.066 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-29T06:51:20.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-29T06:45:20.457527+0000 last_deep_scrub_stamp 2024-03-29T06:51:20.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:51:20.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:20.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-29T06:51:20.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:51:20.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:20.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:51:20.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:20.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:51:20.086 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:20.086 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:51:20.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:20.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:21.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:21.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:21.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:51:21.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:21.396 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:51:21.397 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:21.397 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:51:21.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:21.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:22.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:22.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:22.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:51:22.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:22.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:51:22.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:22.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:51:23.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:23.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:24.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:24.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:24.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:51:24.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:24.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:51:24.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:24.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:51:24.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:24.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:25.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:25.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:25.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:51:25.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:25.349 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:51:25.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:25.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:51:25.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:45:20.457527+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:25.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:26.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:26.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:26.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-29T06:51:26.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:26.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T06:51:26.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:26.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:51:21.276398+0000 '>' 2024-03-29T06:45:20.457527+0000 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-29T06:51:27.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-29T06:51:27.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-03-29T06:51:27.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-03-29T06:51:27.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-03-29T06:51:27.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-03-29T06:51:27.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.058 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.211 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-29T06:51:27.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-03-29T06:51:27.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-29T06:51:27.516 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-29T06:51:27.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-29T06:51:27.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-29T06:51:27.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-29T06:51:27.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-29T06:51:27.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-29T06:51:27.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=212 2024-03-29T06:51:27.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-29T06:51:27.634 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:51:27.634 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-29T06:51:27.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:51:27.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-29T06:51:27.687 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:51:27.687 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:51:27.687 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:51:27.687 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:51:27.687 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:51:27.687 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-29T06:51:27.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:51:27.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:51:27.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:51:27.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-03-29T06:51:27.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-29T06:51:27.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-03-29T06:51:27.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-03-29T06:51:27.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-03-29T06:51:27.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:27.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:51:27.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:27.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:51:28.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:51:21.276398+0000 2024-03-29T06:51:28.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-03-29T06:51:28.368 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0 on osd.1 to repair 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-03-29T06:51:21.276398+0000 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:51:21.276398+0000 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:28.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:51:28.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:28.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:51:28.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:51:21.276398+0000 '>' 2024-03-29T06:51:21.276398+0000 2024-03-29T06:51:28.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:29.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:29.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:29.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:51:29.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:29.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:51:29.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:29.669 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:51:29.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:51:21.276398+0000 '>' 2024-03-29T06:51:21.276398+0000 2024-03-29T06:51:29.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:30.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:30.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:30.975 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:51:30.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:30.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:51:30.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:30.977 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:51:31.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:51:21.276398+0000 '>' 2024-03-29T06:51:21.276398+0000 2024-03-29T06:51:31.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:51:32.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:51:32.285 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:51:32.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:51:32.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:51:32.286 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:51:32.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:51:32.287 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:51:32.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:51:30.089687+0000 '>' 2024-03-29T06:51:21.276398+0000 2024-03-29T06:51:32.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:51:32.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-29T06:51:32.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:51:32.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:51:32.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:51:32.625 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:51:32.625 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:51:32.625 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:51:32.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:51:32.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:51:32.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:51:32.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:51:32.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:51:32.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:51:32.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:51:32.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:51:32.736 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:51:33.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:51:33.092 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:51:33.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:51:33.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:51:33.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:51:33.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=884763262986 2024-03-29T06:51:33.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 884763262986 2024-03-29T06:51:33.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-884763262986' 2024-03-29T06:51:33.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:51:33.274 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:51:33.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=910533066759 2024-03-29T06:51:33.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 910533066759 2024-03-29T06:51:33.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-884763262986 1-910533066759' 2024-03-29T06:51:33.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:51:33.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-884763262986 2024-03-29T06:51:33.418 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:51:33.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:51:33.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-884763262986 2024-03-29T06:51:33.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:51:33.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=884763262986 2024-03-29T06:51:33.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 884763262986' 2024-03-29T06:51:33.422 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 884763262986 2024-03-29T06:51:33.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:51:33.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262985 -lt 884763262986 2024-03-29T06:51:33.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:51:34.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:51:34.727 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:51:35.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 884763262986 -lt 884763262986 2024-03-29T06:51:35.087 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:51:35.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-910533066759 2024-03-29T06:51:35.088 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:51:35.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:51:35.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-910533066759 2024-03-29T06:51:35.090 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:51:35.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=910533066759 2024-03-29T06:51:35.092 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 910533066759 2024-03-29T06:51:35.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 910533066759' 2024-03-29T06:51:35.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:51:35.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 910533066759 -lt 910533066759 2024-03-29T06:51:35.421 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:51:35.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:51:35.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:51:35.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:51:35.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:51:35.811 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:51:35.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:51:35.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:51:35.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:51:35.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:51:35.812 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:51:36.106 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:51:36.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:51:36.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:51:36.107 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:51:36.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:51:36.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:51:36.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:51:36.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-29T06:51:36.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-29T06:51:36.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-29T06:51:36.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-29T06:51:36.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-29T06:51:36.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-29T06:51:36.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-03-29T06:51:36.548 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-29T06:51:37.023 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'csr_pool' removed 2024-03-29T06:51:37.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:51:37.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:51:37.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:51:37.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:51:37.047 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:51:37.047 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:51:37.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:51:37.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:51:37.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:51:37.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:51:37.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:51:37.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:51:37.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:51:37.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:51:37.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:51:37.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:51:37.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:51:37.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:51:37.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:51:37.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:51:37.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:51:37.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:51:37.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:51:37.421 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:51:37.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:37.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:37.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:51:37.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:51:37.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:51:37.424 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:51:37.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:51:37.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:51:37.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:51:37.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:51:37.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:51:37.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:51:37.429 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:51:37.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:51:37.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:51:37.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:51:37.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:51:37.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:51:37.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:51:37.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:51:37.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:51:37.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:51:37.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:51:37.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:51:37.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:37.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:37.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:51:37.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:51:37.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:51:37.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:51:37.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:51:37.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:37.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:37.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:51:37.443 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:51:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:51:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:51:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-29T06:51:37.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:51:37.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:51:37.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:51:37.884 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:51:37.884 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:37.884 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:37.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:51:37.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-29T06:51:38.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:51:38.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:51:38.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:51:38.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:51:38.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:51:38.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:51:38.130 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:51:38.130 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:51:38.130 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:51:38.131 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:38.131 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:38.131 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:38.132 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:51:38.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:51:38.132 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:51:38.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:51:38.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:51:38.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:51:38.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:51:38.249 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:51:38.250 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:51:38.250 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:51:38.250 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:51:38.250 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:51:38.250 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:38.250 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:38.251 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:51:38.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:51:38.251 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:51:38.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-03-29T06:51:38.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:51:38.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:51:38.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:51:38.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:51:38.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:51:38.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:51:40.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:51:40.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:51:40.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:51:40.051 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:51:40.051 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:40.051 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:40.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:51:40.052 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:51:40.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:51:40.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:51:40.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:51:40.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:51:40.095 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:51:40.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d89d499-0cd4-48ed-afbb-54716e5e60f5 2024-03-29T06:51:40.095 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 7d89d499-0cd4-48ed-afbb-54716e5e60f5 2024-03-29T06:51:40.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7d89d499-0cd4-48ed-afbb-54716e5e60f5' 2024-03-29T06:51:40.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:51:40.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8ZAZm/SXqBhAAe0ktUWhFZmk32QM+Kc8row== 2024-03-29T06:51:40.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8ZAZm/SXqBhAAe0ktUWhFZmk32QM+Kc8row=="}' 2024-03-29T06:51:40.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d89d499-0cd4-48ed-afbb-54716e5e60f5 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:51:40.430 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:51:40.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:51:40.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8ZAZm/SXqBhAAe0ktUWhFZmk32QM+Kc8row== --osd-uuid 7d89d499-0cd4-48ed-afbb-54716e5e60f5 2024-03-29T06:51:40.481 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:40.486+0000 7f514c0227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:40.515 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:40.518+0000 7f514c0227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:40.516 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:40.518+0000 7f514c0227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:40.517 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:40.518+0000 7f514c0227c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:51:40.517 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:40.518+0000 7f514c0227c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:51:40.517 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:40.518+0000 7f514c0227c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:51:40.623 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:40.626+0000 7f514c0227c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:51:48.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:51:48.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:51:48.901 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:51:48.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:51:48.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:51:49.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:51:49.297 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:51:49.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:51:49.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:51:49.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:51:49.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:51:49.329 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:49.330+0000 7f9e4abd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:49.378 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:49.382+0000 7f9e4abd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:49.380 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:49.382+0000 7f9e4abd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:51:49.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:51:49.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:51:49.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:51:49.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:51:49.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:51:49.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:49.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:51:49.588 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:51:49.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:49.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:49.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:49.967 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:49.970+0000 7f9e4abd67c0 -1 Falling back to public interface 2024-03-29T06:51:50.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:50.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:50.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:51:50.939 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:51:50.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:50.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:51.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:52.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:52.308 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:52.308 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:51:52.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:51:52.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:52.309 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:52.521 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:51:52.522+0000 7f9e4abd67c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:51:52.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:53.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:53.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:53.589 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:51:53.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:51:53.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:53.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:53.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:54.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:54.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:54.983 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:51:54.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:51:54.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:54.984 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:55.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:56.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:56.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:56.412 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:51:56.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:51:56.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:56.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:56.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:57.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:57.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:57.801 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:51:57.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:51:57.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:57.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:58.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:51:59.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:51:59.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:51:59.116 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:51:59.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:51:59.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:51:59.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:51:59.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:00.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:00.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:00.437 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:52:00.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:52:00.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:00.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:00.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:01.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:01.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:01.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:52:01.973 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:52:01.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:01.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:02.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:03.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:03.378 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:03.379 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:52:03.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:52:03.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:03.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:03.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:04.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:04.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:04.799 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:52:04.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:52:04.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:04.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:05.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:06.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:06.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:06.161 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:52:06.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:52:06.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:06.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:06.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:07.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:07.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:07.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:52:07.681 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:52:07.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:07.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:07.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:08.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:08.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:08.995 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:52:08.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:52:08.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:08.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:09.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:10.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:10.343 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:10.343 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:52:10.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:52:10.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:10.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:10.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:11.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:11.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:11.877 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:52:11.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:52:11.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:11.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:12.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:13.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:13.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:13.195 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:52:13.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:52:13.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:13.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:13.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:14.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:14.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:14.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:52:14.833 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:52:14.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:14.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:15.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:16.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:16.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:16.714 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:52:16.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:52:16.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:16.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:17.808 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:18.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:18.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:18.810 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:52:18.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:52:18.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:18.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:19.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:20.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:20.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:20.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:52:20.128 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:52:20.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:20.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:20.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:21.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:21.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:21.859 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:52:21.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:52:21.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:21.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:22.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:23.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:23.627 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:23.628 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:52:23.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:52:23.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:23.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:24.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:25.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:25.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:25.645 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:52:25.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:52:25.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:25.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:26.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:27.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:27.124 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:27.125 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:52:27.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:52:27.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:27.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:52:27.428 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3601924864,v1:127.0.0.1:6803/3601924864] [v2:127.0.0.1:6804/3601924864,v1:127.0.0.1:6805/3601924864] exists,up 7d89d499-0cd4-48ed-afbb-54716e5e60f5 2024-03-29T06:52:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:52:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:52:27.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:52:27.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:52:27.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:52:27.434 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:52:27.434 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:52:27.434 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:52:27.434 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:52:27.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe22085f-4341-47c0-bdef-c1e2b421ddae 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fe22085f-4341-47c0-bdef-c1e2b421ddae' 2024-03-29T06:52:27.440 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 fe22085f-4341-47c0-bdef-c1e2b421ddae 2024-03-29T06:52:27.441 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:52:27.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQArZQZmNSc9GxAABEQJqGfCQzp4oxTDQMrOrQ== 2024-03-29T06:52:27.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQArZQZmNSc9GxAABEQJqGfCQzp4oxTDQMrOrQ=="}' 2024-03-29T06:52:27.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe22085f-4341-47c0-bdef-c1e2b421ddae -i td/osd-scrub-repair/1/new.json 2024-03-29T06:52:27.858 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:52:27.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:52:27.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArZQZmNSc9GxAABEQJqGfCQzp4oxTDQMrOrQ== --osd-uuid fe22085f-4341-47c0-bdef-c1e2b421ddae 2024-03-29T06:52:27.911 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:27.913+0000 7f6128df57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:52:27.994 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:27.997+0000 7f6128df57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:52:27.996 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:27.997+0000 7f6128df57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:52:27.996 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:27.997+0000 7f6128df57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:52:27.996 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:27.997+0000 7f6128df57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:52:27.997 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:27.997+0000 7f6128df57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:52:28.088 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:28.089+0000 7f6128df57c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:52:36.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:52:36.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:52:36.590 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:52:36.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:52:36.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:52:36.988 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:52:36.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:52:36.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:52:36.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:52:36.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:52:36.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:52:37.020 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:37.021+0000 7f7574eba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:52:37.064 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:37.065+0000 7f7574eba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:52:37.066 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:37.069+0000 7f7574eba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:52:37.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:52:37.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:52:37.290 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:52:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:52:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:52:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:52:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:52:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:37.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:37.627 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:37.629+0000 7f7574eba7c0 -1 Falling back to public interface 2024-03-29T06:52:37.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:38.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:38.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:38.646 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:52:38.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:52:38.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:38.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:38.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:39.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:39.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:39.956 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:52:39.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:52:39.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:39.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:40.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:40.529 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:52:40.529+0000 7f7574eba7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:52:41.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:41.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:41.262 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:52:41.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:52:41.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:41.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:41.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:42.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:42.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:42.623 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:52:42.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:52:42.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:42.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:42.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:43.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:43.974 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:43.980 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:52:43.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:52:43.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:43.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:44.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:45.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:45.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:45.293 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:52:45.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:52:45.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:45.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:45.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:46.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:46.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:46.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:52:46.754 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:52:46.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:46.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:47.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:48.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:48.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:48.122 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:52:48.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:52:48.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:48.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:48.511 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:49.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:49.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:49.514 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:52:49.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:52:49.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:49.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:49.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:50.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:50.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:50.914 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:52:50.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:52:50.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:50.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:51.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:52.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:52.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:52.262 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:52:52.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:52:52.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:52.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:52.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:53.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:53.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:53.575 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:52:53.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:52:53.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:53.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:54.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:55.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:55.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:55.050 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:52:55.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:52:55.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:55.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:55.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:56.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:56.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:56.369 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:52:56.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:52:56.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:56.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:56.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:57.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:57.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:57.876 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:52:57.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:52:57.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:57.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:58.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:52:59.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:52:59.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:52:59.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:52:59.193 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:52:59.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:52:59.194 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:52:59.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:00.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:00.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:00.521 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:53:00.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:53:00.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:00.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:00.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:01.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:01.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:01.895 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:53:01.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:53:01.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:01.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:02.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:03.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:03.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:03.275 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:53:03.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:53:03.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:03.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:04.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:05.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:05.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:05.099 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:53:05.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:53:05.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:05.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:05.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:06.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:06.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:06.651 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:53:06.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:53:06.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:06.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:07.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:08.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:08.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:08.500 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:53:08.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:53:08.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:08.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:08.816 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:09.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:09.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:09.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:53:09.818 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:53:09.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:09.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:10.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:11.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:11.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:11.399 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:53:11.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:53:11.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:11.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:12.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:13.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:13.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:13.180 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:53:13.180 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:53:13.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:13.181 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:14.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:15.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:15.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:15.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:53:15.161 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:53:15.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:15.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:15.561 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1146740206,v1:127.0.0.1:6811/1146740206] [v2:127.0.0.1:6812/1146740206,v1:127.0.0.1:6813/1146740206] exists,up fe22085f-4341-47c0-bdef-c1e2b421ddae 2024-03-29T06:53:15.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:53:15.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:53:15.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:53:15.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-03-29T06:53:15.562 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:53:15.925 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:53:15.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:53:15.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:53:16.393 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:53:16.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:53:17.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:53:20.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-29T06:53:20.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:53:20.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:53:20.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:53:20.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:53:20.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:53:20.073 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:53:20.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:53:20.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:53:20.073 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:53:20.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:53:20.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:53:20.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:53:20.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:53:20.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:53:20.210 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:53:20.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:53:20.526 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:53:20.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:53:20.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:20.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:53:20.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-03-29T06:53:20.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-03-29T06:53:20.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-03-29T06:53:20.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:20.785 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:53:20.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-29T06:53:20.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-29T06:53:20.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2024-03-29T06:53:20.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:20.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-03-29T06:53:20.931 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:20.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:53:20.933 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-03-29T06:53:20.933 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:20.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-03-29T06:53:20.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-03-29T06:53:20.934 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836492 2024-03-29T06:53:20.935 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:21.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-03-29T06:53:21.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:53:22.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:53:22.240 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:22.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-03-29T06:53:22.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:53:23.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T06:53:23.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:23.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836492 2024-03-29T06:53:23.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:23.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:23.849 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-29T06:53:23.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:53:23.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-29T06:53:23.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:23.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-29T06:53:23.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-29T06:53:23.853 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672969 2024-03-29T06:53:23.854 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:53:24.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-03-29T06:53:24.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:53:24.160 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:53:24.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:53:24.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:53:24.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:53:24.593 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:53:24.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:53:24.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:53:24.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:53:24.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:53:24.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:53:24.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:53:24.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:53:24.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:53:24.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:53:25.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:53:25.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:53:25.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:53:25.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-03-29T06:53:25.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-29T06:53:25.853 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'foo' created 2024-03-29T06:53:25.872 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:53:26.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-03-29T06:53:26.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-29T06:53:27.600 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'csr_pool' created 2024-03-29T06:53:27.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:53:28.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-29T06:53:28.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:53:28.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:53:28.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:53:28.626 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:53:28.627 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:53:28.627 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:53:28.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:53:28.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:53:28.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:53:28.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:53:28.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:53:28.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:53:28.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:53:28.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:53:28.769 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:53:29.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:53:29.170 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:53:29.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:53:29.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:29.170 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:53:29.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-03-29T06:53:29.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-03-29T06:53:29.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-03-29T06:53:29.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:29.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:53:29.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-29T06:53:29.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-29T06:53:29.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672971' 2024-03-29T06:53:29.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:29.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-03-29T06:53:29.551 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:29.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:53:29.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-03-29T06:53:29.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:29.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-03-29T06:53:29.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-03-29T06:53:29.555 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836495 2024-03-29T06:53:29.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:29.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-03-29T06:53:29.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:53:30.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:53:30.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:31.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-03-29T06:53:31.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:31.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:31.211 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-29T06:53:31.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:53:31.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-29T06:53:31.214 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:31.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-29T06:53:31.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-29T06:53:31.216 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672971 2024-03-29T06:53:31.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:53:31.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672971 2024-03-29T06:53:31.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:53:31.526 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:53:31.527 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:53:31.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:53:31.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:53:31.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:53:31.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:53:31.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:53:31.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:53:31.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:53:31.913 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:53:32.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:53:32.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:53:32.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:53:32.205 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:53:32.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:53:32.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:53:32.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:53:32.544 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:53:32.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:53:33.160 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:53:33.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:53:33.720 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:53:33.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:53:33.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:53:33.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:53:34.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-29T06:53:34.368 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-29T06:53:34.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-29T06:53:34.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-29T06:53:34.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-29T06:53:34.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:53:34.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-29T06:53:34.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-29T06:53:34.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:53:34.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:53:34.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:53:34.923 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:53:34.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:53:35.416 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:53:35.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:53:35.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:53:35.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:53:35.806 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-29T06:53:36.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-29T06:53:36.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-03-29T06:53:36.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-29T06:53:36.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-29T06:53:36.167 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-29T06:53:36.168 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:53:36.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-03-29T06:53:36.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-03-29T06:53:36.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-29T06:53:36.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-29T06:53:36.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-29T06:53:36.484 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T06:53:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-03-29T06:53:36.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-03-29T06:53:37.090 INFO:tasks.workunit.client.0.smithi123.stdout:created pool csr_pool snap snap1 2024-03-29T06:53:37.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-03-29T06:53:37.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-29T06:53:37.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-29T06:53:37.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-29T06:53:37.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-03-29T06:53:37.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-03-29T06:53:37.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-29T06:53:37.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-29T06:53:37.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:53:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:53:37.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:53:37.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:53:37.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:53:37.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:53:37.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:53:37.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:53:37.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-29T06:53:37.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:53:37.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:53:37.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T06:53:37.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:53:37.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:53:37.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:53:40.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:53:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:53:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:53:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:53:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:53:40.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:53:40.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:53:40.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:53:40.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:53:40.744 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:53:40.744 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:53:40.744 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:53:40.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:53:40.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:53:40.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:53:40.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:53:40.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:53:40.747 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:53:40.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:53:40.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:53:40.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:53:40.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:53:40.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:53:40.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:53:40.780 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:40.783+0000 7f8c9ae2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:53:40.783 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:40.783+0000 7f8c9ae2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:53:40.784 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:40.787+0000 7f8c9ae2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:53:41.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:53:41.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:53:41.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:53:41.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:53:41.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:53:41.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:41.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:53:41.066 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:53:41.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:41.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:41.326 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:41.327+0000 7f8c9ae2d7c0 -1 Falling back to public interface 2024-03-29T06:53:41.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:53:42.427 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:53:42.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:42.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:42.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:43.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:43.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:43.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:53:43.746 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:53:43.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:43.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:44.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:44.339 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:44.339+0000 7f8c9ae2d7c0 -1 osd.1 29 log_to_monitors true 2024-03-29T06:53:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:53:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:53:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:45.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:53:45.399 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 33 up_thru 22 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/2511860227,v1:127.0.0.1:6811/2511860227] [v2:127.0.0.1:6812/2511860227,v1:127.0.0.1:6813/2511860227] exists,up fe22085f-4341-47c0-bdef-c1e2b421ddae 2024-03-29T06:53:45.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:53:45.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:53:45.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:53:45.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:53:45.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:53:45.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:53:45.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:53:45.400 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:53:45.401 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:53:45.401 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:53:45.401 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:53:45.401 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:53:45.401 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:53:45.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:53:45.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:53:45.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:53:45.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:53:45.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:53:45.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:53:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:53:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:53:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:53:45.967 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:45.968 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:53:46.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-29T06:53:46.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-29T06:53:46.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-29T06:53:46.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:46.260 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:53:46.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-03-29T06:53:46.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-03-29T06:53:46.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-141733920770' 2024-03-29T06:53:46.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:46.521 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-29T06:53:46.521 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:46.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:53:46.522 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-29T06:53:46.523 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:46.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-29T06:53:46.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-29T06:53:46.524 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 21474836499 2024-03-29T06:53:46.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:46.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-29T06:53:46.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:53:47.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:53:47.936 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:48.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-29T06:53:48.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:48.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920770 2024-03-29T06:53:48.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:48.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:53:48.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920770 2024-03-29T06:53:48.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:48.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-03-29T06:53:48.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920770' 2024-03-29T06:53:48.270 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 141733920770 2024-03-29T06:53:48.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:53:48.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920770 -lt 141733920770 2024-03-29T06:53:48.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:53:48.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:53:48.620 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:53:48.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:53:48.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:53:48.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:53:48.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:53:48.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:53:48.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:53:48.978 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:53:48.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:53:49.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:53:49.324 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:53:49.324 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:53:49.324 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:53:49.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:53:49.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:53:49.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:53:49.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-29T06:53:49.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-29T06:53:49.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-03-29T06:53:49.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-03-29T06:53:49.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-29T06:53:49.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-03-29T06:53:49.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-29T06:53:49.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T06:53:49.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T06:53:49.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:53:49.862 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:53:49.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:53:49.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:53:49.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:53:50.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:53:53.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:53:53.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:53:53.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:53:53.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:53:53.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:53:53.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:53:53.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:53:53.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:53:53.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:53:53.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:53:53.028 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:53:53.028 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:53:53.028 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:53:53.029 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:53:53.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:53:53.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:53:53.029 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:53:53.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:53:53.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:53:53.031 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:53:53.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:53:53.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:53:53.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:53:53.033 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:53:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:53:53.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:53:53.069 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:53.071+0000 7f487eea77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:53:53.072 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:53.075+0000 7f487eea77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:53:53.073 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:53.075+0000 7f487eea77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:53:53.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:53.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:53:53.606 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:53.607+0000 7f487eea77c0 -1 Falling back to public interface 2024-03-29T06:53:53.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:54.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:54.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:54.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:53:54.682 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:53:54.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:54.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:53:54.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:55.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:55.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:55.987 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:53:55.987 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:53:55.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:55.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:53:56.209 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:53:56.211+0000 7f487eea77c0 -1 osd.0 35 log_to_monitors true 2024-03-29T06:53:56.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:53:57.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:53:57.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:53:57.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:53:57.358 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:53:57.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:53:57.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/4190353239,v1:127.0.0.1:6803/4190353239] [v2:127.0.0.1:6804/4190353239,v1:127.0.0.1:6805/4190353239] exists,up 7d89d499-0cd4-48ed-afbb-54716e5e60f5 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:53:57.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:53:57.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:53:57.819 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:53:57.819 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:53:57.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:53:57.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:53:57.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:53:57.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:53:57.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:53:57.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:53:57.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:53:57.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:53:57.942 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:53:58.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:53:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T06:53:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:53:58.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:58.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:53:58.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-29T06:53:58.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-29T06:53:58.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546' 2024-03-29T06:53:58.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:53:58.537 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:53:58.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920773 2024-03-29T06:53:58.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920773 2024-03-29T06:53:58.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724546 1-141733920773' 2024-03-29T06:53:58.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:58.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724546 2024-03-29T06:53:58.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:58.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:53:58.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724546 2024-03-29T06:53:58.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:58.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-29T06:53:58.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2024-03-29T06:53:58.681 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 167503724546 2024-03-29T06:53:58.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:53:58.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-29T06:53:58.988 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:53:58.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-141733920773 2024-03-29T06:53:58.989 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:53:58.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:53:58.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-141733920773 2024-03-29T06:53:58.993 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:53:58.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920773 2024-03-29T06:53:58.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 141733920773' 2024-03-29T06:53:58.994 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 141733920773 2024-03-29T06:53:58.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:53:59.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920773 2024-03-29T06:53:59.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:54:00.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:54:00.350 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:54:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920773 2024-03-29T06:54:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:54:01.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T06:54:01.671 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:54:01.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920774 -lt 141733920773 2024-03-29T06:54:01.982 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:54:01.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:54:01.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:54:02.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-29T06:54:02.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:54:02.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:54:02.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:54:02.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:54:02.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:54:02.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:54:02.361 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:54:02.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-29T06:54:02.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:54:02.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:54:02.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:54:03.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-29T06:54:03.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:54:03.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:54:03.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-03-29T06:54:03.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-03-29T06:54:03.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-29T06:54:03.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-29T06:54:03.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:54:03.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:54:03.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:54:03.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:54:03.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-29T06:53:27.304781+0000 2024-03-29T06:54:03.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-29T06:54:03.700 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-29T06:54:03.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-29T06:53:27.304781+0000 2024-03-29T06:54:03.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-29T06:54:03.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:53:27.304781+0000 2024-03-29T06:54:03.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:54:03.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:54:03.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:54:03.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:54:03.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:54:03.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:54:03.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:54:03.718 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:54:04.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:53:27.304781+0000 '>' 2024-03-29T06:53:27.304781+0000 2024-03-29T06:54:04.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:54:05.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:54:05.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:54:05.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:54:05.014 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:54:05.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:54:05.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:54:05.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:54:05.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:53:27.304781+0000 '>' 2024-03-29T06:53:27.304781+0000 2024-03-29T06:54:05.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:54:06.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:54:06.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:54:06.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-29T06:54:06.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-29T06:54:06.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:54:06.337 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:54:06.338 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-29T06:54:06.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:54:04.490142+0000 '>' 2024-03-29T06:53:27.304781+0000 2024-03-29T06:54:06.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:54:06.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-03-29T06:54:06.717 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-03-29T06:54:06.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-03-29T06:54:06.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-29T06:54:06.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-03-29T06:54:06.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:54:06.827 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:54:06.828 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-29T06:54:06.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:54:06.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr: . as $in 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr: | if type == "object" then 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr: reduce keys[] as $key 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr: else f 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr: end; 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-29T06:54:06.870 INFO:tasks.workunit.client.0.smithi123.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-29T06:54:06.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-29T06:54:06.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-29T06:54:06.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:54:06.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-29T06:54:06.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-29T06:54:06.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-29T06:54:06.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-29T06:54:06.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-29T06:54:06.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-03-29T06:54:06.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-29T06:54:06.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-29T06:54:06.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-29T06:54:06.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-29T06:54:06.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-29T06:54:06.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-03-29T06:54:06.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-03-29T06:54:06.924 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-29T06:54:07.379 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'csr_pool' removed 2024-03-29T06:54:07.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:54:07.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:54:07.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:54:07.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:54:07.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:54:07.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:54:07.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:54:07.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:54:07.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:54:07.716 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:54:07.716 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:54:07.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:54:07.719 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:54:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:54:07.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:54:07.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:54:07.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:54:07.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:54:07.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:54:07.724 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:54:07.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:54:07.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:54:07.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:54:07.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:54:07.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:07.746 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:07.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:54:07.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:54:07.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:54:07.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:54:07.748 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:54:07.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:54:07.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:54:07.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:54:07.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:54:07.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:54:07.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:54:07.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:54:07.757 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:54:07.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:54:07.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:54:07.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:54:07.759 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:54:07.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:54:07.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:54:07.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:54:07.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:54:07.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:07.763 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:07.763 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:54:07.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:54:07.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:54:07.764 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:54:07.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:54:07.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:07.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:07.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:54:07.768 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:54:07.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:54:07.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:54:07.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:54:07.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T06:54:08.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:54:08.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:54:08.222 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:54:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:54:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:08.223 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:54:08.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T06:54:08.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:54:08.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:54:08.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:54:08.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:54:08.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:54:08.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:54:08.463 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:54:08.463 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:54:08.463 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:54:08.464 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:54:08.464 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:08.464 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:08.464 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:54:08.464 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:54:08.464 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:54:08.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:54:08.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:54:08.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:54:08.582 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:54:08.583 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:54:08.583 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:54:08.583 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:54:08.583 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:54:08.584 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:54:08.584 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:08.584 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:08.584 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:54:08.584 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:54:08.584 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:54:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-29T06:54:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:54:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:54:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:54:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:54:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:54:08.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:54:09.608 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:54:09.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:54:09.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:54:09.609 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:54:09.609 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:09.609 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:09.610 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:54:09.611 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:54:09.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:54:09.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-29T06:54:09.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:54:09.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-29T06:54:09.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:54:09.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:54:09.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:54:09.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:54:09.652 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:54:09.652 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:09.652 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:09.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:54:09.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:54:09.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:54:09.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:54:09.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7969e1a-67a4-43cc-837a-45bb2cf5ac94 2024-03-29T06:54:09.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b7969e1a-67a4-43cc-837a-45bb2cf5ac94' 2024-03-29T06:54:09.655 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 b7969e1a-67a4-43cc-837a-45bb2cf5ac94 2024-03-29T06:54:09.655 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:54:09.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRZQZmAzh5KBAANY0GI7Cajr7PYMNZkctpag== 2024-03-29T06:54:09.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRZQZmAzh5KBAANY0GI7Cajr7PYMNZkctpag=="}' 2024-03-29T06:54:09.674 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7969e1a-67a4-43cc-837a-45bb2cf5ac94 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:54:09.994 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:54:10.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:54:10.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRZQZmAzh5KBAANY0GI7Cajr7PYMNZkctpag== --osd-uuid b7969e1a-67a4-43cc-837a-45bb2cf5ac94 2024-03-29T06:54:10.053 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:10.055+0000 7f4e2aef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:10.095 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:10.095+0000 7f4e2aef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:10.097 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:10.099+0000 7f4e2aef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:10.098 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:10.099+0000 7f4e2aef87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:54:10.098 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:10.099+0000 7f4e2aef87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:54:10.098 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:10.099+0000 7f4e2aef87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:54:10.203 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:10.203+0000 7f4e2aef87c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:54:18.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:54:18.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:54:18.190 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:54:18.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:54:18.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:54:18.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:54:18.565 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:54:18.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:54:18.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:54:18.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:54:18.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:54:18.597 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:18.598+0000 7faafeea07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:18.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:18.642+0000 7faafeea07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:18.646+0000 7faafeea07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:18.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:54:18.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:54:18.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:54:18.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:54:18.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:54:18.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:18.857 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:54:18.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:54:18.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:18.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:19.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:19.230 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:19.230+0000 7faafeea07c0 -1 Falling back to public interface 2024-03-29T06:54:20.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:20.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:20.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:54:20.149 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:54:20.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:20.150 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:20.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:21.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:21.475 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:54:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:54:21.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:21.476 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:21.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:22.104 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:22.106+0000 7faafeea07c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:54:22.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:22.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:22.782 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:54:22.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:54:22.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:22.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:23.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:24.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:24.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:24.129 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:54:24.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:54:24.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:24.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:24.480 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:25.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:25.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:25.482 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:54:25.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:54:25.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:25.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:25.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:26.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:26.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:26.793 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:54:26.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:54:26.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:26.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:27.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:28.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:28.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:28.265 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:54:28.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:54:28.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:28.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:28.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:29.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:29.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:29.586 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:54:29.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:54:29.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:29.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:29.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:30.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:30.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:30.906 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:54:30.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:54:30.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:30.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:31.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:32.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:32.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:32.288 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:54:32.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:54:32.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:32.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:32.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:33.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:33.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:33.606 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:54:33.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:54:33.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:33.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:33.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:34.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:34.985 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:34.985 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:54:34.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:54:34.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:34.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:35.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:36.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:36.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:36.429 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:54:36.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:54:36.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:36.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:36.789 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:37.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:37.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:37.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:54:37.791 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:54:37.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:37.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:38.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:39.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:39.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:39.300 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:54:39.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:54:39.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:39.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:39.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:40.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:40.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:40.617 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:54:40.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:54:40.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:40.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:41.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:42.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:42.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:42.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:54:42.079 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:54:42.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:42.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:43.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:44.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:44.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:44.301 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:54:44.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:54:44.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:44.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:45.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:46.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:46.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:46.018 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:54:46.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:54:46.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:46.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:46.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:47.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:47.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:47.772 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:54:47.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:54:47.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:47.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:48.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:49.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:49.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:49.431 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:54:49.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:54:49.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:49.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:49.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:50.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:50.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:50.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:54:50.847 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:54:50.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:50.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:51.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:52.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:52.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:52.669 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:54:52.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:54:52.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:52.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:53.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:54:54.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:54:54.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:54:54.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:54:54.685 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:54:54.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:54:54.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:54:55.108 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3434724377,v1:127.0.0.1:6803/3434724377] [v2:127.0.0.1:6804/3434724377,v1:127.0.0.1:6805/3434724377] exists,up b7969e1a-67a4-43cc-837a-45bb2cf5ac94 2024-03-29T06:54:55.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:54:55.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:54:55.112 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:54:55.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:54:55.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:54:55.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:54:55.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:54:55.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:54:55.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b957350-3196-4a20-b13c-c19492661257 2024-03-29T06:54:55.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6b957350-3196-4a20-b13c-c19492661257' 2024-03-29T06:54:55.116 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 6b957350-3196-4a20-b13c-c19492661257 2024-03-29T06:54:55.117 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:54:55.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/ZQZmZyCeCBAAdiHV44xmuDtldLWqKiUz9Q== 2024-03-29T06:54:55.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/ZQZmZyCeCBAAdiHV44xmuDtldLWqKiUz9Q=="}' 2024-03-29T06:54:55.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b957350-3196-4a20-b13c-c19492661257 -i td/osd-scrub-repair/1/new.json 2024-03-29T06:54:55.584 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:54:55.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:54:55.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/ZQZmZyCeCBAAdiHV44xmuDtldLWqKiUz9Q== --osd-uuid 6b957350-3196-4a20-b13c-c19492661257 2024-03-29T06:54:55.642 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:55.646+0000 7f4ba0eb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:55.727 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:55.730+0000 7f4ba0eb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:55.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:55.734+0000 7f4ba0eb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:54:55.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:55.734+0000 7f4ba0eb47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:54:55.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:55.734+0000 7f4ba0eb47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:54:55.729 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:55.734+0000 7f4ba0eb47c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:54:55.827 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:54:55.830+0000 7f4ba0eb47c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:55:04.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:55:04.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:55:04.546 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:55:04.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:55:04.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:55:04.947 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:55:04.947 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:55:04.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:55:04.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:55:04.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:55:04.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:55:04.982 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:04.985+0000 7f8097b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:05.015 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:05.021+0000 7f8097b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:05.017 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:05.021+0000 7f8097b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:05.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:55:05.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:55:05.257 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:55:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:55:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:55:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:55:05.258 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:55:05.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:05.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:05.561 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:05.605 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:05.609+0000 7f8097b4b7c0 -1 Falling back to public interface 2024-03-29T06:55:06.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:06.564 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:06.564 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:55:06.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:55:06.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:06.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:06.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:07.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:07.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:07.886 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:55:07.887 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:55:07.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:07.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:08.286 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:08.542 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:08.545+0000 7f8097b4b7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:55:09.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:09.288 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:09.288 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:55:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:55:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:09.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:10.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:10.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:10.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:55:10.681 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:55:10.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:10.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:10.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:11.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:11.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:11.983 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:55:11.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:55:11.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:11.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:12.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:13.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:13.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:13.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:55:13.299 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:55:13.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:13.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:13.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:14.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:14.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:14.761 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:55:14.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:55:14.761 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:14.762 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:15.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:16.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:16.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:16.238 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:55:16.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:55:16.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:16.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:16.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:17.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:17.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:17.635 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:55:17.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:55:17.635 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:17.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:18.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:19.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:19.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:19.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:55:19.078 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:55:19.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:19.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:19.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:20.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:20.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:20.396 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:55:20.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:55:20.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:20.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:20.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:21.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:21.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:21.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:55:21.714 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:55:21.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:21.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:22.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:23.084 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:23.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:23.085 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:55:23.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:55:23.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:23.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:23.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:24.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:24.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:24.404 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:55:24.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:55:24.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:24.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:24.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:25.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:25.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:25.893 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:55:25.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:55:25.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:25.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:26.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:27.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:27.332 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:55:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:55:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:27.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:27.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:28.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:28.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:28.776 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:55:28.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:55:28.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:28.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:29.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:30.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:30.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:30.230 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:55:30.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:55:30.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:30.231 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:31.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:32.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:32.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:32.055 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:55:32.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:55:32.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:32.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:32.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:33.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:33.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:33.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:55:33.566 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:55:33.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:33.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:33.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:34.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:34.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:34.958 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:55:34.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:55:34.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:34.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:35.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:36.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:36.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:36.744 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:55:36.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:55:36.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:36.745 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:37.530 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:38.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:38.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:38.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:55:38.533 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:55:38.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:38.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:39.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:40.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:40.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:40.136 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:55:40.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:55:40.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:40.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:40.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:41.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:41.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:41.445 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:55:41.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:55:41.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:41.446 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:42.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:43.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:43.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:43.320 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:55:43.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:55:43.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:43.321 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:55:43.660 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1749514394,v1:127.0.0.1:6811/1749514394] [v2:127.0.0.1:6812/1749514394,v1:127.0.0.1:6813/1749514394] exists,up 6b957350-3196-4a20-b13c-c19492661257 2024-03-29T06:55:43.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:55:43.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:55:43.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:55:43.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:55:43.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:55:43.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:55:43.667 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:55:43.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88444bf9-d9c5-4851-9d73-15774e8d3b4e 2024-03-29T06:55:43.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 88444bf9-d9c5-4851-9d73-15774e8d3b4e' 2024-03-29T06:55:43.668 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 88444bf9-d9c5-4851-9d73-15774e8d3b4e 2024-03-29T06:55:43.668 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:55:43.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvZQZmIy5sKRAA3Bk26i/ji/pUfcoWj9s6+g== 2024-03-29T06:55:43.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvZQZmIy5sKRAA3Bk26i/ji/pUfcoWj9s6+g=="}' 2024-03-29T06:55:43.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88444bf9-d9c5-4851-9d73-15774e8d3b4e -i td/osd-scrub-repair/2/new.json 2024-03-29T06:55:44.244 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:55:44.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T06:55:44.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvZQZmIy5sKRAA3Bk26i/ji/pUfcoWj9s6+g== --osd-uuid 88444bf9-d9c5-4851-9d73-15774e8d3b4e 2024-03-29T06:55:44.301 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:44.305+0000 7f73e37c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:44.388 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:44.393+0000 7f73e37c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:44.390 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:44.393+0000 7f73e37c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:44.390 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:44.393+0000 7f73e37c67c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:55:44.390 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:44.393+0000 7f73e37c67c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:55:44.391 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:44.393+0000 7f73e37c67c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:55:44.485 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:44.489+0000 7f73e37c67c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T06:55:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T06:55:53.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:55:53.066 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T06:55:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T06:55:53.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:55:53.454 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T06:55:53.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T06:55:53.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:55:53.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:55:53.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:55:53.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:55:53.486 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:53.488+0000 7fb5bf07e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:53.488 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:53.492+0000 7fb5bf07e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:53.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:53.492+0000 7fb5bf07e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:55:53.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T06:55:53.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:55:53.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:55:53.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:55:53.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:55:53.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:53.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:55:53.741 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:55:53.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:53.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:55:54.057 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:54.060+0000 7fb5bf07e7c0 -1 Falling back to public interface 2024-03-29T06:55:54.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:55.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:55.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:55.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:55:55.079 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:55:55.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:55.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:55:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:56.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:56.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:56.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:55:56.398 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:55:56.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:56.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:55:56.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:57.224 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:55:57.228+0000 7fb5bf07e7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T06:55:57.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:57.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:55:57.711 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:55:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:57.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:55:58.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:55:59.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:55:59.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:55:59.296 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:55:59.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:55:59.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:55:59.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:55:59.769 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:00.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:00.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:00.771 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:56:00.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:56:00.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:00.772 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:01.182 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:02.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:02.184 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:02.185 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:56:02.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:56:02.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:02.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:02.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:03.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:03.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:03.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:56:03.743 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:56:03.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:03.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:04.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:05.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:05.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:05.176 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:56:05.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:56:05.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:05.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:05.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:06.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:06.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:06.506 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:56:06.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:56:06.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:06.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:06.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:07.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:07.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:07.845 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:56:07.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:56:07.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:07.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:08.160 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:09.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:09.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:09.162 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:56:09.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:56:09.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:09.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:09.607 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:10.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:10.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:10.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:56:10.610 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:56:10.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:10.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:11.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:12.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:12.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:12.145 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:56:12.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:56:12.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:12.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:12.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:13.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:13.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:13.732 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:56:13.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:56:13.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:13.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:14.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:15.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:15.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:15.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:56:15.047 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:56:15.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:15.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:15.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:16.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:16.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:16.355 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:56:16.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:56:16.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:16.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:16.829 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:17.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:17.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:17.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:56:17.831 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:56:17.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:17.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:18.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:19.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:19.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:19.137 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:56:19.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:56:19.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:19.138 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:19.581 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:20.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:20.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:20.583 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:56:20.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:56:20.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:20.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:21.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:22.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:22.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:22.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:56:22.498 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:56:22.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:22.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:23.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:24.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:24.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:24.223 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:56:24.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:56:24.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:24.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:24.540 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:25.542 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:25.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:25.543 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:56:25.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:56:25.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:25.543 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:26.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:27.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:27.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:27.074 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:56:27.074 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:56:27.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:27.075 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:27.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:28.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:28.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:28.415 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:56:28.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:56:28.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:28.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:29.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:30.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:30.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:30.410 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:56:30.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:56:30.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:30.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:30.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:56:31.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:56:31.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:56:31.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:56:31.736 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:56:31.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:56:31.736 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:56:32.051 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1838249029,v1:127.0.0.1:6819/1838249029] [v2:127.0.0.1:6820/1838249029,v1:127.0.0.1:6821/1838249029] exists,up 88444bf9-d9c5-4851-9d73-15774e8d3b4e 2024-03-29T06:56:32.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:56:32.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:56:32.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:56:32.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-29T06:56:32.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:56:32.465 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T06:56:32.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:56:32.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:56:32.973 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:56:32.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:56:33.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:56:37.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:56:37.253 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:56:37.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:56:37.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:56:37.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:56:37.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:56:37.388 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:56:37.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:56:37.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:56:37.705 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:56:37.705 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:56:37.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:56:37.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:56:37.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:56:37.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-29T06:56:37.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-29T06:56:37.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-29T06:56:37.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:56:37.852 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:56:37.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-29T06:56:37.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-29T06:56:37.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-03-29T06:56:37.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:56:37.986 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:56:38.110 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-29T06:56:38.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-29T06:56:38.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509448' 2024-03-29T06:56:38.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:56:38.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-29T06:56:38.111 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:56:38.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:56:38.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-29T06:56:38.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:56:38.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-29T06:56:38.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-29T06:56:38.115 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836506 2024-03-29T06:56:38.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:56:38.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-29T06:56:38.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:56:38.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-29T06:56:38.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:56:38.455 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:56:38.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-29T06:56:38.456 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:56:38.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-29T06:56:38.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-29T06:56:38.458 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672979 2024-03-29T06:56:38.458 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:56:38.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-03-29T06:56:38.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:56:38.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-29T06:56:38.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:56:38.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:56:38.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-29T06:56:38.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:56:38.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-29T06:56:38.814 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509448 2024-03-29T06:56:38.815 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-29T06:56:38.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:56:39.148 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-29T06:56:39.148 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:56:39.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:56:39.149 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:56:39.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T06:56:39.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:56:39.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:56:39.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:56:39.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:56:39.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:56:39.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:56:39.556 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:56:39.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T06:56:39.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:56:39.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:56:39.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T06:56:40.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-29T06:56:40.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T06:56:40.840 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T06:56:42.014 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T06:56:42.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:56:43.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-29T06:56:43.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T06:56:43.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:56:43.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:56:43.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:56:43.037 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:56:43.038 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:56:43.038 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:56:43.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:56:43.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:56:43.039 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:56:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:56:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:56:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:56:43.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:56:43.178 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:56:43.178 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:56:43.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:56:43.750 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:56:43.750 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:56:43.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:56:43.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:56:43.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:56:43.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-29T06:56:43.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-29T06:56:43.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-29T06:56:43.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:56:43.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:56:44.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-29T06:56:44.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-29T06:56:44.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-03-29T06:56:44.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:56:44.116 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:56:44.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-29T06:56:44.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-29T06:56:44.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509450' 2024-03-29T06:56:44.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:56:44.328 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-29T06:56:44.328 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:56:44.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:56:44.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-29T06:56:44.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:56:44.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-29T06:56:44.332 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836508 2024-03-29T06:56:44.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-29T06:56:44.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:56:44.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-29T06:56:44.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:56:44.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-29T06:56:44.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:56:44.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:56:44.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-29T06:56:44.657 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:56:44.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-29T06:56:44.658 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672982 2024-03-29T06:56:44.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-29T06:56:44.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:56:45.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-03-29T06:56:45.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:56:45.005 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-29T06:56:45.006 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:56:45.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:56:45.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-29T06:56:45.008 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:56:45.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-29T06:56:45.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-29T06:56:45.010 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509450 2024-03-29T06:56:45.010 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:56:45.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509450 2024-03-29T06:56:45.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:56:46.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:56:46.372 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:56:46.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-29T06:56:46.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:56:46.695 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:56:46.696 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:56:47.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:56:47.064 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:56:47.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:56:47.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:56:47.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:56:47.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:56:47.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:56:47.065 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:56:47.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:56:47.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:56:47.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:56:47.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:56:47.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:56:47.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:56:47.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:56:47.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T06:56:47.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-29T06:56:47.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:56:47.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:56:47.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-29T06:56:47.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:56:47.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:56:47.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:56:48.301 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:56:48.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:56:48.853 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:56:48.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:56:48.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:56:48.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:56:49.227 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-29T06:56:49.227 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T06:56:49.228 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-29T06:56:49.228 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-29T06:56:49.228 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:56:49.613 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T06:56:49.614 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T06:56:50.090 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T06:56:50.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T06:56:50.617 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T06:56:50.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T06:56:50.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T06:56:50.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-29T06:56:51.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-29T06:56:51.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T06:56:51.049 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-29T06:56:51.049 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-29T06:56:51.050 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T06:56:51.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-29T06:56:51.382 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:56:51.382 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T06:56:51.382 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-29T06:56:51.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-29T06:56:51.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-29T06:56:51.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-29T06:56:51.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:56:51.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T06:56:51.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:56:51.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:56:51.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:56:51.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:56:51.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:56:52.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:56:52.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:56:52.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T06:56:52.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:56:52.698 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:56:52.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:56:52.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:56:52.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:56:54.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:56:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:56:54.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T06:56:54.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:56:54.016 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:56:54.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:56:54.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:56:54.016 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:56:55.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:56:55.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-29T06:56:55.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-29T06:56:57.095 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-29T06:56:57.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-29T06:56:57.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-29T06:56:59.474 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-29T06:57:00.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-29T06:57:00.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-29T06:57:01.379 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-29T06:57:01.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-29T06:57:01.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-29T06:57:03.706 INFO:tasks.workunit.client.0.smithi123.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-29T06:57:04.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-29T06:57:04.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:57:04.243 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:57:04.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:57:04.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:57:04.248 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:57:04.248 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:04.248 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:04.248 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:04.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:04.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:57:04.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:57:04.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:57:04.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T06:57:04.250 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T06:57:04.250 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:57:04.250 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T06:57:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T06:57:04.252 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:57:04.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:57:04.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:57:04.284 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:04.287+0000 7f05088597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:04.286 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:04.291+0000 7f05088597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:04.288 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:04.291+0000 7f05088597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:04.574 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:04.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:04.840 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:04.843+0000 7f05088597c0 -1 Falling back to public interface 2024-03-29T06:57:04.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:05.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:05.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:05.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:57:05.928 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:57:05.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:05.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:06.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:07.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:07.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:07.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:57:07.277 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:57:07.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:07.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:07.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:07.846 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:07.847+0000 7f05088597c0 -1 osd.0 33 log_to_monitors true 2024-03-29T06:57:08.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:08.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:08.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:57:08.598 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:57:08.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:08.599 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:09.135 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 40 up_thru 17 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/1311706978,v1:127.0.0.1:6803/1311706978] [v2:127.0.0.1:6804/1311706978,v1:127.0.0.1:6805/1311706978] exists,up b7969e1a-67a4-43cc-837a-45bb2cf5ac94 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:57:09.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:57:09.137 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:57:09.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:57:09.140 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T06:57:09.141 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T06:57:09.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:57:09.142 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T06:57:09.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T06:57:09.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:57:09.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:57:09.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:57:09.169 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:09.171+0000 7f1d6a7097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:09.258 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:09.263+0000 7f1d6a7097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:09.260 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:09.263+0000 7f1d6a7097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:09.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:57:09.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:09.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:57:09.791 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:09.815 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:09.819+0000 7f1d6a7097c0 -1 Falling back to public interface 2024-03-29T06:57:10.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:10.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:10.792 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:57:10.792 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:57:10.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:10.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:57:11.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:12.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:12.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:12.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:57:12.097 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:57:12.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:12.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:57:12.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:12.683 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:12.687+0000 7f1d6a7097c0 -1 osd.1 35 log_to_monitors true 2024-03-29T06:57:13.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:13.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:13.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:57:13.500 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:57:13.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:13.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:57:13.819 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 43 up_thru 34 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/3066679108,v1:127.0.0.1:6811/3066679108] [v2:127.0.0.1:6812/3066679108,v1:127.0.0.1:6813/3066679108] exists,up 6b957350-3196-4a20-b13c-c19492661257 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:57:13.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:57:13.821 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:13.822 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:13.822 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T06:57:13.824 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.2 2024-03-29T06:57:13.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:57:13.825 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T06:57:13.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T06:57:13.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T06:57:13.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T06:57:13.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T06:57:13.848 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:13.851+0000 7f73ae6517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:13.850 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:13.855+0000 7f73ae6517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:13.851 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:13.855+0000 7f73ae6517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:14.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:57:14.415 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:14.419+0000 7f73ae6517c0 -1 Falling back to public interface 2024-03-29T06:57:14.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:15.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:15.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:15.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:57:15.870 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:57:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:15.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:57:16.195 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:17.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:17.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:17.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:57:17.197 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T06:57:17.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:17.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:57:17.463 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:17.466+0000 7f73ae6517c0 -1 osd.2 37 log_to_monitors true 2024-03-29T06:57:17.517 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:18.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:18.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:18.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:57:18.519 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T06:57:18.519 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:18.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:57:19.020 INFO:tasks.workunit.client.0.smithi123.stderr:osd.2 up in weight 1 up_from 47 up_thru 36 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/4177311361,v1:127.0.0.1:6819/4177311361] [v2:127.0.0.1:6820/4177311361,v1:127.0.0.1:6821/4177311361] exists,up 88444bf9-d9c5-4851-9d73-15774e8d3b4e 2024-03-29T06:57:19.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:57:19.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:57:19.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:57:19.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-29T06:57:19.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T06:57:20.334 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' removed 2024-03-29T06:57:20.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T06:57:20.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T06:57:21.205 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T06:57:21.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T06:57:22.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T06:57:24.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T06:57:24.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T06:57:24.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T06:57:24.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T06:57:24.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T06:57:24.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T06:57:24.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T06:57:24.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T06:57:24.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T06:57:25.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T06:57:25.136 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T06:57:25.136 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T06:57:25.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T06:57:25.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:57:25.136 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T06:57:25.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-29T06:57:25.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-29T06:57:25.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-03-29T06:57:25.284 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:57:25.284 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T06:57:25.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593732 2024-03-29T06:57:25.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593732 2024-03-29T06:57:25.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-184683593732' 2024-03-29T06:57:25.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T06:57:25.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T06:57:25.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-03-29T06:57:25.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-03-29T06:57:25.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-184683593732 2-201863462915' 2024-03-29T06:57:25.563 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:57:25.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-03-29T06:57:25.564 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:57:25.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T06:57:25.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-03-29T06:57:25.566 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:57:25.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-29T06:57:25.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-03-29T06:57:25.568 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 171798691845 2024-03-29T06:57:25.568 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:57:25.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-03-29T06:57:25.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T06:57:26.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T06:57:26.868 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T06:57:27.189 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-29T06:57:27.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:57:27.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593732 2024-03-29T06:57:27.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:57:27.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T06:57:27.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593732 2024-03-29T06:57:27.195 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:57:27.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593732 2024-03-29T06:57:27.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593732' 2024-03-29T06:57:27.197 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 184683593732 2024-03-29T06:57:27.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T06:57:27.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593732 2024-03-29T06:57:27.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T06:57:27.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-03-29T06:57:27.550 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T06:57:27.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T06:57:27.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-03-29T06:57:27.553 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T06:57:27.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-03-29T06:57:27.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-03-29T06:57:27.555 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 201863462915 2024-03-29T06:57:27.555 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T06:57:27.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-03-29T06:57:27.905 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T06:57:27.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:57:27.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:57:28.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T06:57:28.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T06:57:28.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T06:57:28.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T06:57:28.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T06:57:28.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T06:57:28.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T06:57:28.282 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T06:57:28.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T06:57:28.617 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T06:57:28.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T06:57:28.618 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T06:57:28.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T06:57:28.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T06:57:28.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T06:57:28.966 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-29T06:57:28.966 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T06:57:28.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-29T06:57:28.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-29T06:57:28.967 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T06:57:29.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-29T06:57:29.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-29T06:57:29.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T06:57:29.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T06:57:29.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:57:29.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:57:29.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:57:29.281 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:57:29.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:29.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T06:57:29.890 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-29T06:57:29.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:29.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T06:57:29.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:29.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T06:57:29.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T06:57:29.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:57:29.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:57:29.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:57:29.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:57:29.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:57:29.907 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:57:30.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:56:41.812494+0000 '>' 2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:30.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:57:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:57:31.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:57:31.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:57:31.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:57:31.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:57:31.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:57:31.193 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:57:31.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:56:41.812494+0000 '>' 2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:31.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:57:32.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:57:32.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:57:32.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:57:32.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:57:32.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:57:32.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:57:32.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:57:32.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:56:41.812494+0000 '>' 2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:32.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:57:33.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:57:33.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:57:33.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:57:33.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:57:33.803 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:57:33.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:57:33.804 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:57:34.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:56:41.812494+0000 '>' 2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:34.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T06:57:35.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T06:57:35.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T06:57:35.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T06:57:35.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T06:57:35.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T06:57:35.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T06:57:35.122 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T06:57:35.419 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T06:57:30.392859+0000 '>' 2024-03-29T06:56:41.812494+0000 2024-03-29T06:57:35.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T06:57:35.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-29T06:57:35.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-29T06:57:35.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-29T06:57:35.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-29T06:57:35.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-29T06:57:35.422 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-29T06:57:35.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-29T06:57:35.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:57:35.570 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:57:35.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:57:35.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:57:35.883 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:57:35.884 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:57:35.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:57:35.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:57:35.886 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:57:35.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:57:35.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:57:35.888 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:57:35.889 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:57:35.890 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:57:35.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:57:35.892 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:57:35.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:57:35.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:35.917 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:35.917 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T06:57:35.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T06:57:35.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T06:57:35.920 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T06:57:35.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T06:57:35.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T06:57:35.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T06:57:35.923 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T06:57:35.924 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T06:57:35.925 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T06:57:35.925 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T06:57:35.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T06:57:35.927 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T06:57:35.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T06:57:35.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:57:35.928 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T06:57:35.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T06:57:35.929 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T06:57:35.930 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T06:57:35.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T06:57:35.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T06:57:35.934 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T06:57:35.934 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:35.934 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:35.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T06:57:35.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T06:57:35.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T06:57:35.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T06:57:35.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T06:57:35.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:35.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:35.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T06:57:35.939 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T06:57:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T06:57:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T06:57:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T06:57:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-03-29T06:57:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-03-29T06:57:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-03-29T06:57:35.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T06:57:35.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T06:57:36.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T06:57:36.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:57:36.383 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:57:36.384 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:57:36.384 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:36.384 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:36.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:36.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T06:57:36.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T06:57:36.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T06:57:36.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:57:36.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:57:36.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T06:57:36.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T06:57:36.605 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:57:36.606 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:57:36.606 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:57:36.606 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:57:36.607 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:36.607 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:36.607 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:57:36.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:57:36.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T06:57:36.720 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T06:57:36.720 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T06:57:36.720 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T06:57:36.720 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T06:57:36.721 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T06:57:36.721 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T06:57:36.721 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T06:57:36.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T06:57:36.721 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T06:57:36.722 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:36.722 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:36.722 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T06:57:36.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T06:57:36.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T06:57:36.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-29T06:57:36.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T06:57:36.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T06:57:36.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T06:57:36.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T06:57:36.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T06:57:36.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T06:57:37.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T06:57:37.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:57:37.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:57:37.775 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:57:37.775 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:37.775 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:37.775 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:37.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:57:37.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T06:57:37.813 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-29T06:57:37.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:57:37.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-29T06:57:37.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:57:37.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:57:37.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T06:57:37.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:57:37.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:57:37.818 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:57:37.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:57:37.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:57:37.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:57:37.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T06:57:37.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:57:37.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4661a23-d956-4393-a962-cdb0c5633c99 2024-03-29T06:57:37.820 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b4661a23-d956-4393-a962-cdb0c5633c99' 2024-03-29T06:57:37.821 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 b4661a23-d956-4393-a962-cdb0c5633c99 2024-03-29T06:57:37.821 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:57:37.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhZgZmArJgMhAAMXnkBewboDnzSoNnXIC/Lw== 2024-03-29T06:57:37.837 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhZgZmArJgMhAAMXnkBewboDnzSoNnXIC/Lw=="}' 2024-03-29T06:57:37.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4661a23-d956-4393-a962-cdb0c5633c99 -i td/osd-scrub-repair/0/new.json 2024-03-29T06:57:38.131 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:57:38.147 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T06:57:38.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhZgZmArJgMhAAMXnkBewboDnzSoNnXIC/Lw== --osd-uuid b4661a23-d956-4393-a962-cdb0c5633c99 2024-03-29T06:57:38.182 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:38.190+0000 7f88538167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:38.215 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:38.222+0000 7f88538167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:38.217 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:38.222+0000 7f88538167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:38.217 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:38.226+0000 7f88538167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:57:38.218 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:38.226+0000 7f88538167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:57:38.218 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:38.226+0000 7f88538167c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T06:57:38.324 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:38.330+0000 7f88538167c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T06:57:46.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T06:57:46.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:57:46.474 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T06:57:46.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T06:57:46.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:57:46.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T06:57:46.862 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T06:57:46.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:57:46.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:57:46.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:57:46.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:57:46.895 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:46.902+0000 7f0df82dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:46.937 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:46.942+0000 7f0df82dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:46.938 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:46.946+0000 7f0df82dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:57:47.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T06:57:47.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:57:47.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T06:57:47.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:57:47.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:57:47.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:47.169 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:57:47.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:57:47.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:47.170 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:47.466 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:47.540 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:47.546+0000 7f0df82dc7c0 -1 Falling back to public interface 2024-03-29T06:57:48.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:48.467 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:48.468 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:57:48.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:57:48.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:48.468 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:48.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:49.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:49.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:49.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:57:49.786 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:57:49.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:49.786 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:50.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:50.459 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:57:50.466+0000 7f0df82dc7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T06:57:51.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:51.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:51.106 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:57:51.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:57:51.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:51.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:51.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:52.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:52.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:52.631 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:57:52.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:57:52.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:52.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:52.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:53.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:53.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:53.953 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:57:53.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:57:53.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:53.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:54.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:55.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:55.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:55.344 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:57:55.345 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:57:55.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:55.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:55.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:56.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:56.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:56.853 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:57:56.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:57:56.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:56.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:57.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:58.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:58.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:58.157 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:57:58.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:57:58.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:58.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:58.553 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:57:59.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:57:59.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:57:59.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:57:59.555 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:57:59.555 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:57:59.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:57:59.952 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:00.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:00.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:00.954 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:58:00.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:58:00.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:00.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:01.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:02.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:02.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:02.275 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:58:02.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:58:02.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:02.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:02.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:03.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:03.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:03.754 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:58:03.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:58:03.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:03.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:04.069 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:05.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:05.071 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:58:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:58:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:05.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:05.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:06.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:06.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:06.390 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:58:06.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:58:06.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:06.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:06.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:07.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:07.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:07.896 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:58:07.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:58:07.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:07.897 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:08.258 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:09.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:09.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:09.260 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:58:09.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:58:09.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:09.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:09.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:10.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:10.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:10.645 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:58:10.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:58:10.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:10.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:11.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:12.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:12.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:12.120 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:58:12.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:58:12.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:12.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:12.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:13.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:13.877 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:13.878 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:58:13.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:58:13.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:13.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:14.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:15.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:15.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:15.478 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:58:15.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:58:15.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:15.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:15.796 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:16.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:16.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:16.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:58:16.799 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:58:16.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:16.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:17.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:18.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:18.107 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:18.107 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:58:18.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:58:18.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:18.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:19.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:20.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:20.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:20.020 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:58:20.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:58:20.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:20.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:20.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:21.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:21.671 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:21.672 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:58:21.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:58:21.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:21.673 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:22.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:23.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:23.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:23.352 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:58:23.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:58:23.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:23.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:23.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:24.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:24.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:24.951 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T06:58:24.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T06:58:24.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:24.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T06:58:25.263 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3690682115,v1:127.0.0.1:6803/3690682115] [v2:127.0.0.1:6804/3690682115,v1:127.0.0.1:6805/3690682115] exists,up b4661a23-d956-4393-a962-cdb0c5633c99 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T06:58:25.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:58:25.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:58:25.268 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:58:25.268 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:58:25.268 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:58:25.268 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:58:25.270 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:58:25.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:58:25.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T06:58:25.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:58:25.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b934b0bc-4105-4593-a6e2-a88fedb30d34 2024-03-29T06:58:25.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b934b0bc-4105-4593-a6e2-a88fedb30d34' 2024-03-29T06:58:25.272 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 b934b0bc-4105-4593-a6e2-a88fedb30d34 2024-03-29T06:58:25.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:58:25.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRZgZmuj3dERAARseOPsJKHzKM3OrfhaE/yQ== 2024-03-29T06:58:25.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRZgZmuj3dERAARseOPsJKHzKM3OrfhaE/yQ=="}' 2024-03-29T06:58:25.292 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b934b0bc-4105-4593-a6e2-a88fedb30d34 -i td/osd-scrub-repair/1/new.json 2024-03-29T06:58:25.690 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:58:25.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T06:58:25.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRZgZmuj3dERAARseOPsJKHzKM3OrfhaE/yQ== --osd-uuid b934b0bc-4105-4593-a6e2-a88fedb30d34 2024-03-29T06:58:25.739 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:25.745+0000 7f12ad5707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:58:25.825 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:25.829+0000 7f12ad5707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:58:25.826 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:25.833+0000 7f12ad5707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:58:25.826 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:25.833+0000 7f12ad5707c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:58:25.827 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:25.833+0000 7f12ad5707c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:58:25.827 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:25.833+0000 7f12ad5707c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T06:58:25.915 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:25.921+0000 7f12ad5707c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T06:58:34.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T06:58:34.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:58:34.495 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T06:58:34.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T06:58:34.496 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:58:34.879 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T06:58:34.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T06:58:34.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:58:34.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:58:34.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:58:34.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:58:34.912 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:34.917+0000 7f1ae1d407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:58:34.920 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:34.925+0000 7f1ae1d407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:58:34.921 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:34.929+0000 7f1ae1d407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:58:35.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T06:58:35.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:58:35.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T06:58:35.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:58:35.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:58:35.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:35.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:58:35.199 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:58:35.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:35.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:35.500 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:35.505+0000 7f1ae1d407c0 -1 Falling back to public interface 2024-03-29T06:58:35.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:36.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:36.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:36.526 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:58:36.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:58:36.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:36.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:36.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:37.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:37.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:37.839 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:58:37.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:58:37.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:37.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:38.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:38.484 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:58:38.489+0000 7f1ae1d407c0 -1 osd.1 0 log_to_monitors true 2024-03-29T06:58:39.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:39.144 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:39.145 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:58:39.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:58:39.145 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:39.146 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:39.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:40.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:40.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:40.701 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:58:40.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:58:40.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:40.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:41.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:42.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:42.118 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:42.118 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:58:42.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:58:42.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:42.119 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:42.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:43.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:43.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:43.463 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:58:43.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:58:43.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:43.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:43.898 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:44.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:44.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:44.900 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:58:44.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:58:44.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:44.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:45.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:46.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:46.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:46.374 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:58:46.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:58:46.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:46.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:47.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:48.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:48.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:48.042 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:58:48.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:58:48.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:48.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:48.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:49.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:49.447 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:49.447 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:58:49.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:58:49.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:49.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:49.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:50.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:50.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:50.867 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:58:50.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:58:50.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:50.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:51.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:52.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:52.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:52.357 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:58:52.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:58:52.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:52.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:52.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:53.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:53.851 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:53.851 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:58:53.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:58:53.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:53.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:54.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:55.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:55.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:55.161 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:58:55.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:58:55.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:55.162 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:55.469 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:56.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:56.471 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:56.471 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:58:56.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:58:56.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:56.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:57.039 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:58.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:58.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:58.041 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:58:58.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:58:58.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:58.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:58:58.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:58:59.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:58:59.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:58:59.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:58:59.433 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:58:59.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:58:59.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:00.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:01.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:01.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:01.116 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:59:01.116 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:59:01.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:01.117 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:02.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:03.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:03.085 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:03.085 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:59:03.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:59:03.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:03.086 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:03.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:04.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:04.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:04.631 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:59:04.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:59:04.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:04.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:05.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:06.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:06.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:06.024 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:59:06.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:59:06.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:06.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:07.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:08.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:08.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:08.082 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:59:08.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:59:08.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:08.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:08.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:09.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:09.730 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:09.731 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:59:09.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:59:09.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:09.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:10.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:11.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:11.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:11.494 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:59:11.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:59:11.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:11.495 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:11.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:12.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:12.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:12.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:59:12.830 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:59:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:12.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T06:59:13.195 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/230115809,v1:127.0.0.1:6811/230115809] [v2:127.0.0.1:6812/230115809,v1:127.0.0.1:6813/230115809] exists,up b934b0bc-4105-4593-a6e2-a88fedb30d34 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T06:59:13.198 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T06:59:13.199 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T06:59:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T06:59:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T06:59:13.200 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:59:13.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T06:59:13.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T06:59:13.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T06:59:13.202 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-29T06:59:13.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T06:59:13.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T06:59:13.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=194fba3c-3cc4-4e22-982d-50f7e4ab4164 2024-03-29T06:59:13.203 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 194fba3c-3cc4-4e22-982d-50f7e4ab4164' 2024-03-29T06:59:13.203 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 194fba3c-3cc4-4e22-982d-50f7e4ab4164 2024-03-29T06:59:13.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T06:59:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBZgZm4wKSDRAAGIOw8ii92qOPnNfgxlJMAQ== 2024-03-29T06:59:13.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBZgZm4wKSDRAAGIOw8ii92qOPnNfgxlJMAQ=="}' 2024-03-29T06:59:13.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 194fba3c-3cc4-4e22-982d-50f7e4ab4164 -i td/osd-scrub-repair/2/new.json 2024-03-29T06:59:13.614 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:59:13.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T06:59:13.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBZgZm4wKSDRAAGIOw8ii92qOPnNfgxlJMAQ== --osd-uuid 194fba3c-3cc4-4e22-982d-50f7e4ab4164 2024-03-29T06:59:13.667 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:13.672+0000 7fde009797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:59:13.749 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:13.752+0000 7fde009797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:59:13.751 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:13.756+0000 7fde009797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:59:13.751 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:13.756+0000 7fde009797c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:59:13.751 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:13.756+0000 7fde009797c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:59:13.751 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:13.756+0000 7fde009797c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T06:59:13.838 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:13.844+0000 7fde009797c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T06:59:22.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T06:59:22.221 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T06:59:22.221 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T06:59:22.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T06:59:22.222 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T06:59:22.597 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T06:59:22.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T06:59:22.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T06:59:22.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T06:59:22.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T06:59:22.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T06:59:22.629 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:22.631+0000 7fe15ff1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:59:22.630 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:22.635+0000 7fe15ff1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:59:22.632 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:22.635+0000 7fe15ff1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T06:59:22.894 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T06:59:22.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T06:59:22.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T06:59:22.895 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T06:59:22.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T06:59:22.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T06:59:22.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:22.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T06:59:22.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:22.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:23.164 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:23.167+0000 7fe15ff1c7c0 -1 Falling back to public interface 2024-03-29T06:59:23.205 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:24.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:24.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:24.207 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T06:59:24.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T06:59:24.207 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:24.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:24.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:25.522 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:25.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:25.523 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T06:59:25.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T06:59:25.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:25.523 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:25.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:26.148 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T06:59:26.151+0000 7fe15ff1c7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T06:59:26.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:26.843 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:26.843 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T06:59:26.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T06:59:26.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:26.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:27.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:28.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:28.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:28.381 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T06:59:28.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T06:59:28.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:28.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:28.864 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:29.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:29.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:29.866 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T06:59:29.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T06:59:29.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:29.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:30.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:31.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:31.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:31.341 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T06:59:31.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T06:59:31.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:31.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:31.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:32.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:32.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:32.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T06:59:32.645 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T06:59:32.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:32.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:32.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:33.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:33.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:33.971 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T06:59:33.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T06:59:33.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:33.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:34.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:35.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:35.329 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:35.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T06:59:35.330 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T06:59:35.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:35.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:35.739 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:36.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:36.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:36.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T06:59:36.742 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T06:59:36.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:36.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:37.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:38.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:38.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:38.051 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T06:59:38.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T06:59:38.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:38.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:38.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:39.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:39.369 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:39.369 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T06:59:39.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T06:59:39.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:39.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:39.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:40.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:40.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:40.694 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T06:59:40.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T06:59:40.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:40.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:41.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:42.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:42.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:42.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T06:59:42.014 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T06:59:42.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:42.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:42.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:43.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:43.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:43.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T06:59:43.374 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T06:59:43.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:43.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:43.695 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:44.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:44.697 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T06:59:44.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T06:59:44.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:44.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:45.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:46.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:46.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:46.017 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T06:59:46.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T06:59:46.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:46.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:46.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:47.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:47.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:47.507 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T06:59:47.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T06:59:47.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:47.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:47.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:48.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:48.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:48.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T06:59:48.862 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T06:59:48.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:48.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:50.009 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:51.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:51.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:51.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T06:59:51.011 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T06:59:51.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:51.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:51.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:52.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:52.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:52.751 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T06:59:52.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T06:59:52.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:52.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:53.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:54.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:54.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:54.175 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T06:59:54.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T06:59:54.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:54.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:54.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:55.854 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:55.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:55.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T06:59:55.855 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T06:59:55.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:55.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:56.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:57.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:57.306 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:57.307 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T06:59:57.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T06:59:57.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:57.307 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:57.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T06:59:58.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T06:59:58.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T06:59:58.717 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T06:59:58.717 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T06:59:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T06:59:58.718 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T06:59:59.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:00.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:00.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:00.624 INFO:tasks.workunit.client.0.smithi123.stdout:26 2024-03-29T07:00:00.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 26 2024-03-29T07:00:00.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:00.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:00:00.933 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1568703739,v1:127.0.0.1:6819/1568703739] [v2:127.0.0.1:6820/1568703739,v1:127.0.0.1:6821/1568703739] exists,up 194fba3c-3cc4-4e22-982d-50f7e4ab4164 2024-03-29T07:00:00.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:00:00.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:00:00.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:00:00.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-29T07:00:00.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T07:00:01.307 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T07:00:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T07:00:01.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T07:00:01.780 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T07:00:01.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T07:00:02.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T07:00:05.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-29T07:00:05.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:00:05.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:00:05.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:00:05.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:00:05.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:00:05.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:00:05.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:00:05.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:00:05.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:00:06.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:00:06.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:00:06.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:00:06.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:00:06.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:00:06.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:00:06.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:00:06.341 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:06.341 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T07:00:06.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:00:06.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:06.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:00:06.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-29T07:00:06.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-29T07:00:06.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-29T07:00:06.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:06.490 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:00:06.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705681 2024-03-29T07:00:06.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705681 2024-03-29T07:00:06.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-38654705681' 2024-03-29T07:00:06.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:06.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T07:00:06.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542157 2024-03-29T07:00:06.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542157 2024-03-29T07:00:06.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-38654705681 2-60129542157' 2024-03-29T07:00:06.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:06.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-29T07:00:06.776 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:06.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:00:06.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-29T07:00:06.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:06.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-29T07:00:06.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-29T07:00:06.779 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836509 2024-03-29T07:00:06.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:00:07.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-29T07:00:07.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:00:08.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:00:08.136 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:00:08.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-29T07:00:08.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:00:09.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T07:00:09.445 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:00:09.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-29T07:00:09.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:09.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:09.751 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705681 2024-03-29T07:00:09.753 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:00:09.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705681 2024-03-29T07:00:09.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:09.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705681 2024-03-29T07:00:09.755 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 38654705681 2024-03-29T07:00:09.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705681' 2024-03-29T07:00:09.756 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:00:10.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705682 -lt 38654705681 2024-03-29T07:00:10.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:10.061 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542157 2024-03-29T07:00:10.061 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:10.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T07:00:10.064 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:10.064 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542157 2024-03-29T07:00:10.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542157 2024-03-29T07:00:10.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2024-03-29T07:00:10.065 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 60129542157 2024-03-29T07:00:10.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:00:10.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542157 -lt 60129542157 2024-03-29T07:00:10.376 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:00:10.377 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:00:10.377 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:00:10.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T07:00:10.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:00:10.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:00:10.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:00:10.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:00:10.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:00:10.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:00:10.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:00:11.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T07:00:11.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:00:11.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:00:11.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:00:11.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T07:00:11.566 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:00:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:00:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-29T07:00:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-29T07:00:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T07:00:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-29T07:00:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T07:00:11.567 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-29T07:00:12.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-29T07:00:12.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-29T07:00:12.953 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'ecpool' created 2024-03-29T07:00:12.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T07:00:13.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-29T07:00:13.982 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-29T07:00:14.894 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 2 allow_ec_overwrites to true 2024-03-29T07:00:14.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T07:00:14.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:00:14.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:00:14.920 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:00:14.921 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:00:14.922 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:00:14.922 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:00:14.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:00:14.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:00:14.923 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:00:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:00:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:00:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:00:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:00:15.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:00:15.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:00:15.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:00:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T07:00:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:00:15.444 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:15.444 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:00:15.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-03-29T07:00:15.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-03-29T07:00:15.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-03-29T07:00:15.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:15.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:00:15.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705684 2024-03-29T07:00:15.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705684 2024-03-29T07:00:15.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-38654705684' 2024-03-29T07:00:15.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:15.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T07:00:16.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542160 2024-03-29T07:00:16.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542160 2024-03-29T07:00:16.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-38654705684 2-60129542160' 2024-03-29T07:00:16.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:16.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-03-29T07:00:16.098 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:16.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:00:16.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-03-29T07:00:16.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:16.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-03-29T07:00:16.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-03-29T07:00:16.102 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836512 2024-03-29T07:00:16.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:00:16.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836512 2024-03-29T07:00:16.403 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:00:17.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:00:17.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:00:17.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-03-29T07:00:17.719 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:17.720 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705684 2024-03-29T07:00:17.720 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:17.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:00:17.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705684 2024-03-29T07:00:17.723 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:17.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705684 2024-03-29T07:00:17.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705684' 2024-03-29T07:00:17.725 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 38654705684 2024-03-29T07:00:17.725 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:00:18.108 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705684 -lt 38654705684 2024-03-29T07:00:18.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:18.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:18.109 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-60129542160 2024-03-29T07:00:18.111 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T07:00:18.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:18.112 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-60129542160 2024-03-29T07:00:18.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542160 2024-03-29T07:00:18.114 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 60129542160 2024-03-29T07:00:18.114 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2024-03-29T07:00:18.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:00:18.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542160 -lt 60129542160 2024-03-29T07:00:18.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:00:18.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:00:18.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:00:18.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T07:00:18.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:00:18.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:00:18.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:00:18.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:00:18.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:00:18.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:00:18.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:00:19.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T07:00:19.076 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:00:19.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:00:19.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:00:19.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T07:00:19.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T07:00:19.845 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T07:00:19.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T07:00:20.489 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T07:00:20.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T07:00:20.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T07:00:20.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:00:21.064 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-29T07:00:21.064 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T07:00:21.064 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-29T07:00:21.064 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-29T07:00:21.064 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T07:00:21.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-29T07:00:21.427 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:21.427 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T07:00:21.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-29T07:00:21.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-29T07:00:21.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-29T07:00:21.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-29T07:00:21.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T07:00:21.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-29T07:00:21.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-29T07:00:21.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-29T07:00:21.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-29T07:00:21.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-29T07:00:21.909 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is set 2024-03-29T07:00:21.931 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-29T07:00:22.511 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T07:00:22.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T07:00:22.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T07:00:22.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:00:23.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-29T07:00:23.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-29T07:00:23.138 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-29T07:00:23.139 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-29T07:00:23.139 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-29T07:00:23.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-29T07:00:23.461 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:23.461 INFO:tasks.workunit.client.0.smithi123.stderr:0' 2024-03-29T07:00:23.461 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-29T07:00:23.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-29T07:00:23.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-29T07:00:23.462 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-29T07:00:23.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T07:00:23.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T07:00:23.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:00:23.463 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:00:23.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:00:23.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:00:23.464 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:00:23.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:00:23.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T07:00:23.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T07:00:23.775 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:00:23.775 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:00:23.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:00:23.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:00:23.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:00:25.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:00:25.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T07:00:25.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-29T07:00:25.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:00:25.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:00:25.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:00:25.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:00:25.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:00:26.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:00:26.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-29T07:00:26.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-29T07:00:28.129 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-29T07:00:28.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-29T07:00:28.662 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-29T07:00:30.492 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-29T07:00:31.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-29T07:00:31.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-29T07:00:32.103 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-29T07:00:32.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-29T07:00:32.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-29T07:00:34.473 INFO:tasks.workunit.client.0.smithi123.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-29T07:00:35.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-29T07:00:35.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T07:00:35.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:00:35.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:00:35.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:00:35.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:00:35.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:00:35.027 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T07:00:35.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T07:00:35.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T07:00:35.028 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T07:00:35.028 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T07:00:35.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T07:00:35.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T07:00:35.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:00:35.030 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:00:35.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:00:35.062 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:35.069+0000 7fa80bf007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:35.065 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:35.073+0000 7fa80bf007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:35.066 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:35.073+0000 7fa80bf007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:35.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T07:00:35.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:00:35.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T07:00:35.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:00:35.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:00:35.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:35.347 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:00:35.348 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T07:00:35.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:35.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:00:35.596 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:35.601+0000 7fa80bf007c0 -1 Falling back to public interface 2024-03-29T07:00:35.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:36.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:36.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:36.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:00:36.725 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:36.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:36.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:00:37.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:38.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:38.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:38.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:00:38.045 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T07:00:38.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:38.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:00:38.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:38.406 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:38.413+0000 7fa80bf007c0 -1 osd.0 33 log_to_monitors true 2024-03-29T07:00:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:00:39.360 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T07:00:39.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:39.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:00:39.682 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/1753356290,v1:127.0.0.1:6803/1753356290] [v2:127.0.0.1:6804/1753356290,v1:127.0.0.1:6805/1753356290] exists,up b4661a23-d956-4393-a962-cdb0c5633c99 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:00:39.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:00:39.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:00:39.686 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:00:39.686 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:00:39.686 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:00:39.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:00:39.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:00:39.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:00:39.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T07:00:39.687 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.1 2024-03-29T07:00:39.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T07:00:39.688 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T07:00:39.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T07:00:39.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:00:39.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:00:39.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:00:39.718 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:39.725+0000 7f432234f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:39.851 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:39.857+0000 7f432234f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:39.853 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:39.861+0000 7f432234f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:40.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T07:00:40.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:00:40.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T07:00:40.071 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:00:40.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:00:40.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:40.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:00:40.072 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T07:00:40.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:40.072 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:00:40.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:40.418 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:40.425+0000 7f432234f7c0 -1 Falling back to public interface 2024-03-29T07:00:41.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:41.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:41.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:00:41.384 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:41.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:41.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:00:41.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:42.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:42.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:42.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:00:42.688 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T07:00:42.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:42.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:00:42.989 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:43.314 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:43.321+0000 7f432234f7c0 -1 osd.1 34 log_to_monitors true 2024-03-29T07:00:43.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:43.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:43.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:00:43.991 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T07:00:43.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:43.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:00:44.520 INFO:tasks.workunit.client.0.smithi123.stderr:osd.1 up in weight 1 up_from 43 up_thru 34 down_at 35 last_clean_interval [9,34) [v2:127.0.0.1:6810/2689582852,v1:127.0.0.1:6811/2689582852] [v2:127.0.0.1:6812/2689582852,v1:127.0.0.1:6813/2689582852] exists,up b934b0bc-4105-4593-a6e2-a88fedb30d34 2024-03-29T07:00:44.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:00:44.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:00:44.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:00:44.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-29T07:00:44.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:00:44.521 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:00:44.522 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:00:44.522 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:00:44.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.2 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-29T07:00:44.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-29T07:00:44.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-29T07:00:44.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:00:44.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:00:44.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:00:44.555 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:44.561+0000 7f4fefa077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:44.557 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:44.565+0000 7f4fefa077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:44.558 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:44.565+0000 7f4fefa077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:00:45.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:45.012 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:00:45.118 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:45.125+0000 7f4fefa077c0 -1 Falling back to public interface 2024-03-29T07:00:45.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:46.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:46.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:46.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:00:46.526 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:46.526 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:46.527 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:00:46.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:47.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:47.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:47.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:00:47.838 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T07:00:47.838 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:47.839 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:00:48.000 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:00:48.005+0000 7f4fefa077c0 -1 osd.2 37 log_to_monitors true 2024-03-29T07:00:48.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:00:49.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:00:49.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:00:49.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:00:49.159 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T07:00:49.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:00:49.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:00:49.668 INFO:tasks.workunit.client.0.smithi123.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [14,37) [v2:127.0.0.1:6818/3083942437,v1:127.0.0.1:6819/3083942437] [v2:127.0.0.1:6820/3083942437,v1:127.0.0.1:6821/3083942437] exists,up 194fba3c-3cc4-4e22-982d-50f7e4ab4164 2024-03-29T07:00:49.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:00:49.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:00:49.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:00:49.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-29T07:00:49.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T07:00:50.883 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' removed 2024-03-29T07:00:50.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T07:00:50.907 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T07:00:51.853 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T07:00:51.876 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T07:00:52.878 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T07:00:55.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-29T07:00:55.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:00:55.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:00:55.741 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:00:55.742 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:00:55.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:00:55.743 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:00:55.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:00:55.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:00:55.744 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:00:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:00:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:00:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:00:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:00:55.880 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:00:55.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:00:56.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:00:56.193 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:00:56.193 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T07:00:56.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:00:56.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:56.194 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:00:56.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-29T07:00:56.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-29T07:00:56.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-03-29T07:00:56.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:56.392 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:00:56.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593732 2024-03-29T07:00:56.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593732 2024-03-29T07:00:56.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-184683593732' 2024-03-29T07:00:56.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:00:56.559 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T07:00:56.698 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-03-29T07:00:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-03-29T07:00:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-184683593732 2-201863462915' 2024-03-29T07:00:56.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:56.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-03-29T07:00:56.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:56.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:00:56.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-03-29T07:00:56.702 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:56.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-29T07:00:56.704 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 171798691845 2024-03-29T07:00:56.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-03-29T07:00:56.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:00:57.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-29T07:00:57.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:57.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:57.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593732 2024-03-29T07:00:57.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:00:57.005 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593732 2024-03-29T07:00:57.005 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:57.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593732 2024-03-29T07:00:57.006 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 184683593732 2024-03-29T07:00:57.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593732' 2024-03-29T07:00:57.007 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:00:57.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593731 -lt 184683593732 2024-03-29T07:00:57.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:00:58.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:00:58.355 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:00:58.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593732 2024-03-29T07:00:58.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:00:58.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:00:58.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-03-29T07:00:58.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T07:00:58.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-03-29T07:00:58.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:00:58.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-03-29T07:00:58.683 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 201863462915 2024-03-29T07:00:58.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-03-29T07:00:58.683 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:00:58.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462916 -lt 201863462915 2024-03-29T07:00:58.998 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:00:58.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:00:58.999 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:00:59.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T07:00:59.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:00:59.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:00:59.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:00:59.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:00:59.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:00:59.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:00:59.394 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:00:59.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T07:00:59.770 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:00:59.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:00:59.771 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:01:00.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T07:01:00.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:01:00.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:01:00.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-29T07:01:00.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-29T07:01:00.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-29T07:01:00.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-29T07:01:00.124 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T07:01:00.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-29T07:01:00.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-29T07:01:00.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-29T07:01:00.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-29T07:01:00.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:01:00.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:01:00.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:01:00.485 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:01:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T07:00:12.769655+0000 2024-03-29T07:01:00.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-29T07:01:01.110 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-29T07:01:01.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-29T07:00:12.769655+0000 2024-03-29T07:01:01.125 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T07:01:01.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T07:00:12.769655+0000 2024-03-29T07:01:01.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T07:01:01.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T07:01:01.126 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:01:01.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:01:01.126 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:01:01.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:01:01.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:01:01.127 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:01:01.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:00:12.769655+0000 '>' 2024-03-29T07:00:12.769655+0000 2024-03-29T07:01:01.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:01:02.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:01:02.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:01:02.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:01:02.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:01:02.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:01:02.431 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:01:02.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:01:02.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:00:12.769655+0000 '>' 2024-03-29T07:00:12.769655+0000 2024-03-29T07:01:02.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:01:03.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:01:03.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:01:03.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:01:03.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:01:03.733 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:01:03.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:01:03.734 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:01:04.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:01:01.812800+0000 '>' 2024-03-29T07:00:12.769655+0000 2024-03-29T07:01:04.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T07:01:04.052 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-29T07:01:04.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-29T07:01:04.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-29T07:01:04.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-29T07:01:04.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-29T07:01:04.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-29T07:01:04.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-29T07:01:04.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-29T07:01:04.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-29T07:01:04.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T07:01:04.217 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T07:01:04.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T07:01:04.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T07:01:04.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:01:04.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:01:04.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:01:04.218 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:01:04.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:01:04.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:01:04.328 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T07:01:04.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T07:01:04.329 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T07:01:04.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T07:01:04.330 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T07:01:04.330 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T07:01:04.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:01:04.331 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T07:01:04.332 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T07:01:04.332 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:01:04.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:01:04.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T07:01:04.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T07:01:04.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T07:01:04.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:04.353 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:04.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T07:01:04.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T07:01:04.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T07:01:04.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T07:01:04.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T07:01:04.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T07:01:04.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T07:01:04.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T07:01:04.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T07:01:04.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T07:01:04.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:01:04.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:01:04.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:01:04.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:01:04.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:01:04.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:01:04.358 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T07:01:04.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T07:01:04.359 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T07:01:04.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T07:01:04.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T07:01:04.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T07:01:04.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:01:04.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T07:01:04.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T07:01:04.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:01:04.362 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:01:04.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T07:01:04.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T07:01:04.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T07:01:04.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:04.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:04.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T07:01:04.365 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T07:01:04.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T07:01:04.366 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T07:01:04.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T07:01:04.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:04.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:04.367 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-29T07:01:04.370 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T07:01:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T07:01:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T07:01:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T07:01:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T07:01:04.371 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-29T07:01:04.830 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T07:01:04.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:01:04.831 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:01:04.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:01:04.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:04.831 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:04.832 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:01:04.832 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-29T07:01:05.083 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T07:01:05.084 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T07:01:05.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T07:01:05.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T07:01:05.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T07:01:05.085 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T07:01:05.085 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T07:01:05.085 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T07:01:05.086 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T07:01:05.086 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T07:01:05.086 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:05.086 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:05.087 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T07:01:05.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T07:01:05.087 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T07:01:05.203 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T07:01:05.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T07:01:05.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T07:01:05.204 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T07:01:05.205 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T07:01:05.206 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T07:01:05.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-29T07:01:05.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T07:01:05.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T07:01:05.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T07:01:05.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T07:01:05.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T07:01:05.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T07:01:06.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T07:01:06.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:01:06.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:01:06.217 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:01:06.217 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:06.218 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:06.218 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:01:06.219 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T07:01:06.219 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T07:01:06.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:01:06.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:01:06.263 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:01:06.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:01:06.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:01:06.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T07:01:06.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-29T07:01:06.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T07:01:06.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T07:01:06.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=463465d0-7d60-46a6-9177-f117ac627042 2024-03-29T07:01:06.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 463465d0-7d60-46a6-9177-f117ac627042' 2024-03-29T07:01:06.266 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 463465d0-7d60-46a6-9177-f117ac627042 2024-03-29T07:01:06.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T07:01:06.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyZwZmXydcERAAIYa5h+EbbI+uxnmEiNGcZw== 2024-03-29T07:01:06.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyZwZmXydcERAAIYa5h+EbbI+uxnmEiNGcZw=="}' 2024-03-29T07:01:06.283 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 463465d0-7d60-46a6-9177-f117ac627042 -i td/osd-scrub-repair/0/new.json 2024-03-29T07:01:06.603 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:01:06.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T07:01:06.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAyZwZmXydcERAAIYa5h+EbbI+uxnmEiNGcZw== --osd-uuid 463465d0-7d60-46a6-9177-f117ac627042 2024-03-29T07:01:06.656 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:06.661+0000 7f643063f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:06.686 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:06.693+0000 7f643063f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:06.688 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:06.693+0000 7f643063f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:06.689 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:06.697+0000 7f643063f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:01:06.689 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:06.697+0000 7f643063f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:01:06.689 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:06.697+0000 7f643063f7c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:01:06.786 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:06.793+0000 7f643063f7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T07:01:14.732 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T07:01:14.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T07:01:14.734 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T07:01:14.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T07:01:14.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T07:01:15.128 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T07:01:15.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T07:01:15.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-29T07:01:15.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T07:01:15.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T07:01:15.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T07:01:15.164 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:15.168+0000 7f801b9e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:15.208 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:15.212+0000 7f801b9e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:15.210 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:15.216+0000 7f801b9e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T07:01:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:01:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T07:01:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:01:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:01:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:01:15.416 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:01:15.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:15.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:15.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:15.794 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:15.800+0000 7f801b9e07c0 -1 Falling back to public interface 2024-03-29T07:01:16.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:16.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:16.726 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:01:16.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:01:16.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:16.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:17.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:18.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:18.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:18.044 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:01:18.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:01:18.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:18.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:18.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:18.805 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:18.808+0000 7f801b9e07c0 -1 osd.0 0 log_to_monitors true 2024-03-29T07:01:19.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:19.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:19.353 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:01:19.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:01:19.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:19.354 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:19.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:20.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:20.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:20.703 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T07:01:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T07:01:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:21.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:22.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:22.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:22.098 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T07:01:22.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T07:01:22.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:22.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:22.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:23.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:23.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:23.416 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T07:01:23.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T07:01:23.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:23.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:23.893 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:24.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:24.895 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:24.895 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T07:01:24.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T07:01:24.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:24.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:25.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:26.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:26.212 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:26.212 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T07:01:26.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T07:01:26.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:26.213 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:26.598 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:27.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:27.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:27.600 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T07:01:27.600 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T07:01:27.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:27.601 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:27.939 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:28.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:28.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:28.941 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T07:01:28.942 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T07:01:28.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:28.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:29.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:30.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:30.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:30.264 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T07:01:30.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T07:01:30.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:30.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:30.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:31.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:31.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:31.578 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T07:01:31.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T07:01:31.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:31.579 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:32.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:33.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:33.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:33.080 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T07:01:33.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T07:01:33.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:33.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:33.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:34.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:34.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:34.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T07:01:34.429 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T07:01:34.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:34.429 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:34.919 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:35.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:35.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:35.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T07:01:35.921 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T07:01:35.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:35.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:36.235 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:37.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:37.237 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T07:01:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T07:01:37.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:37.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:37.699 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:38.700 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:38.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:38.701 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T07:01:38.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T07:01:38.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:38.702 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:39.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:40.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:40.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:40.411 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T07:01:40.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T07:01:40.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:40.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:40.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:41.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:41.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:41.800 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T07:01:41.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T07:01:41.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:41.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:42.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:43.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:43.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:43.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T07:01:43.644 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T07:01:43.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:43.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:44.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:45.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:45.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:45.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T07:01:45.156 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T07:01:45.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:45.157 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:46.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:47.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:47.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:47.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T07:01:47.193 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T07:01:47.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:47.193 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:47.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:49.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:49.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:49.001 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T07:01:49.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T07:01:49.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:49.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:49.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:50.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:50.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:50.755 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T07:01:50.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T07:01:50.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:50.755 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:51.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:01:52.224 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:01:52.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:01:52.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T07:01:52.225 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T07:01:52.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:01:52.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:01:52.643 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2650688469,v1:127.0.0.1:6803/2650688469] [v2:127.0.0.1:6804/2650688469,v1:127.0.0.1:6805/2650688469] exists,up 463465d0-7d60-46a6-9177-f117ac627042 2024-03-29T07:01:52.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:01:52.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:01:52.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:01:52.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-29T07:01:52.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T07:01:52.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:01:52.645 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T07:01:52.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:01:52.646 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T07:01:52.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=495bee18-3b9a-4158-bc12-622b3254dd50 2024-03-29T07:01:52.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 495bee18-3b9a-4158-bc12-622b3254dd50' 2024-03-29T07:01:52.650 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 495bee18-3b9a-4158-bc12-622b3254dd50 2024-03-29T07:01:52.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T07:01:52.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgZwZmJR2FKBAA2V7Dxp5H1e4pl7Iiq99eMg== 2024-03-29T07:01:52.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgZwZmJR2FKBAA2V7Dxp5H1e4pl7Iiq99eMg=="}' 2024-03-29T07:01:52.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 495bee18-3b9a-4158-bc12-622b3254dd50 -i td/osd-scrub-repair/1/new.json 2024-03-29T07:01:53.294 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:01:53.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T07:01:53.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBgZwZmJR2FKBAA2V7Dxp5H1e4pl7Iiq99eMg== --osd-uuid 495bee18-3b9a-4158-bc12-622b3254dd50 2024-03-29T07:01:53.358 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:53.363+0000 7f379bfa57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:53.438 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:53.443+0000 7f379bfa57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:53.440 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:53.443+0000 7f379bfa57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:01:53.441 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:53.447+0000 7f379bfa57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:01:53.441 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:53.447+0000 7f379bfa57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:01:53.441 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:53.447+0000 7f379bfa57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:01:53.532 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:01:53.535+0000 7f379bfa57c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T07:02:01.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T07:02:01.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T07:02:01.779 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T07:02:01.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T07:02:01.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T07:02:02.163 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T07:02:02.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T07:02:02.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-29T07:02:02.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T07:02:02.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T07:02:02.166 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T07:02:02.197 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:02:02.203+0000 7fdfc81007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:02:02.234 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:02:02.239+0000 7fdfc81007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:02:02.235 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:02:02.239+0000 7fdfc81007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:02:02.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T07:02:02.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:02:02.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T07:02:02.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:02:02.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:02:02.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:02.473 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:02:02.473 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:02:02.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:02.474 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:02.768 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:02.831 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:02:02.835+0000 7fdfc81007c0 -1 Falling back to public interface 2024-03-29T07:02:03.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:03.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:03.771 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:02:03.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:02:03.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:03.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:04.077 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:05.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:05.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:05.079 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:02:05.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:02:05.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:05.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:05.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:05.920 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:02:05.923+0000 7fdfc81007c0 -1 osd.1 0 log_to_monitors true 2024-03-29T07:02:06.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:06.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:06.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:02:06.401 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:02:06.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:06.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:07.008 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:08.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:08.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:08.010 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T07:02:08.011 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T07:02:08.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:08.011 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:08.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:09.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:09.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:09.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T07:02:09.462 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T07:02:09.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:09.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:09.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:10.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:10.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:10.971 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T07:02:10.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T07:02:10.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:10.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:11.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:12.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:12.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:12.439 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T07:02:12.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T07:02:12.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:12.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:12.749 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:13.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:13.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:13.751 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T07:02:13.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T07:02:13.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:13.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:14.206 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:15.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:15.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:15.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T07:02:15.209 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T07:02:15.209 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:15.210 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:15.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:16.531 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:16.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:16.532 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T07:02:16.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T07:02:16.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:16.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:16.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:17.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:17.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:17.858 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T07:02:17.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T07:02:17.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:17.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:18.381 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:19.382 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:19.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:19.383 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T07:02:19.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T07:02:19.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:19.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:19.701 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:20.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:20.703 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:20.703 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T07:02:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T07:02:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:20.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:21.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:22.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:22.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:22.340 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T07:02:22.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T07:02:22.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:22.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:22.654 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:23.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:23.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:23.656 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T07:02:23.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T07:02:23.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:23.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:23.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:24.993 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:24.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:24.994 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T07:02:24.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T07:02:24.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:24.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:25.459 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:26.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:26.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:26.461 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T07:02:26.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T07:02:26.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:26.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:27.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:28.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:28.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:28.023 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T07:02:28.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T07:02:28.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:28.024 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:28.337 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:29.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:29.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:29.339 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T07:02:29.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T07:02:29.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:29.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:30.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:31.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:31.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:31.229 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T07:02:31.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T07:02:31.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:31.230 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:32.420 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:33.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:33.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:33.423 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T07:02:33.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T07:02:33.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:33.423 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:34.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:35.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:35.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:35.270 INFO:tasks.workunit.client.0.smithi123.stdout:22 2024-03-29T07:02:35.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-29T07:02:35.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:35.271 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:36.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:37.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:37.105 INFO:tasks.workunit.client.0.smithi123.stdout:23 2024-03-29T07:02:37.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-03-29T07:02:37.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:37.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:37.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:38.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:38.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:38.944 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 24 2024-03-29T07:02:38.945 INFO:tasks.workunit.client.0.smithi123.stdout:24 2024-03-29T07:02:38.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:38.945 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:39.626 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:02:40.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:02:40.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:02:40.870 INFO:tasks.workunit.client.0.smithi123.stdout:25 2024-03-29T07:02:40.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 25 2024-03-29T07:02:40.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:02:40.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:02:40.948 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1350899700,v1:127.0.0.1:6811/1350899700] [v2:127.0.0.1:6812/1350899700,v1:127.0.0.1:6813/1350899700] exists,up 495bee18-3b9a-4158-bc12-622b3254dd50 2024-03-29T07:02:40.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:02:40.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:02:40.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:02:40.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-03-29T07:02:40.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-29T07:02:41.239 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' does not exist 2024-03-29T07:02:41.255 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-29T07:02:41.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-29T07:02:41.704 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'rbd' created 2024-03-29T07:02:41.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T07:02:42.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-29T07:02:45.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-29T07:02:45.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:02:45.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:02:45.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:02:45.983 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:02:45.984 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:02:45.984 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:02:45.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:02:45.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:02:45.985 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:02:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:02:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:02:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:02:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:02:46.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:02:46.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:02:46.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:02:46.450 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:02:46.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:02:46.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:02:46.451 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:02:46.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-29T07:02:46.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-29T07:02:46.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-29T07:02:46.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:02:46.641 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:02:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-29T07:02:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-29T07:02:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672970' 2024-03-29T07:02:46.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:02:46.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-29T07:02:46.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:02:46.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:02:46.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-29T07:02:46.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:02:46.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-29T07:02:46.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-29T07:02:46.782 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836497 2024-03-29T07:02:46.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:02:47.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-29T07:02:47.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:02:47.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-29T07:02:47.095 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:02:47.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:02:47.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-29T07:02:47.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:02:47.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-29T07:02:47.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-29T07:02:47.099 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672970 2024-03-29T07:02:47.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:02:47.524 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-29T07:02:47.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:02:47.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:02:47.525 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:02:47.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-29T07:02:47.975 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:02:47.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:02:47.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:02:47.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:02:47.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:02:47.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:02:47.976 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:02:48.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-29T07:02:48.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:02:48.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:02:48.276 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:02:48.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-29T07:02:48.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:02:48.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:02:48.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-03-29T07:02:48.632 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-03-29T07:02:49.249 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'psr_pool' created 2024-03-29T07:02:49.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T07:02:50.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-29T07:02:50.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:02:50.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:02:50.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:02:50.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:02:50.277 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:02:50.277 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:02:50.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:02:50.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:02:50.277 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:02:50.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:02:50.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:02:50.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:02:50.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:02:50.417 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:02:50.417 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:02:50.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:02:50.780 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:02:50.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:02:50.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:02:50.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:02:50.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-29T07:02:50.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-29T07:02:50.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-29T07:02:50.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:02:50.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:02:51.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-29T07:02:51.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-29T07:02:51.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672972' 2024-03-29T07:02:51.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:02:51.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-29T07:02:51.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:02:51.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:02:51.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-29T07:02:51.103 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:02:51.104 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-29T07:02:51.105 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836499 2024-03-29T07:02:51.105 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-29T07:02:51.105 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:02:51.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-29T07:02:51.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:02:52.392 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:02:52.393 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:02:52.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-29T07:02:52.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:02:53.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-29T07:02:53.699 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:02:54.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-29T07:02:54.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:02:54.021 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-29T07:02:54.022 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:02:54.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:02:54.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-29T07:02:54.024 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:02:54.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-29T07:02:54.026 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672972 2024-03-29T07:02:54.026 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-29T07:02:54.026 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:02:54.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-29T07:02:54.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:02:54.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:02:54.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:02:54.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T07:02:54.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:02:54.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:02:54.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:02:54.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:02:54.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:02:54.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:02:54.713 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:02:54.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T07:02:54.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:02:54.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:02:54.996 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-03-29T07:02:55.395 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-03-29T07:02:55.935 INFO:tasks.workunit.client.0.smithi123.stderr:noscrub is unset 2024-03-29T07:02:55.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-03-29T07:02:56.469 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is unset 2024-03-29T07:02:56.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-29T07:02:56.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-29T07:02:56.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-03-29T07:02:56.869 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-03-29T07:02:56.869 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-03-29T07:02:56.869 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-03-29T07:02:56.870 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-03-29T07:02:56.870 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T07:02:57.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-03-29T07:02:57.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-03-29T07:02:57.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-03-29T07:02:57.262 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-03-29T07:02:57.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-03-29T07:02:57.263 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T07:02:57.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-03-29T07:02:57.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-03-29T07:02:57.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-03-29T07:02:57.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T07:02:57.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-29T07:02:57.663 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:02:57.664 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:02:57.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:02:57.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:02:57.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:02:58.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:02:58.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T07:02:58.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:02:58.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:02:58.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:02:58.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:02:58.976 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:02:58.977 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-29T07:03:01.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-29T07:03:01.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:03:01.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:03:01.372 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:03:01.373 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:03:01.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:03:01.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:03:01.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:03:01.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-29T07:03:01.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:start osd.0 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:03:01.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:03:01.411 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:01.418+0000 7ff868d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:01.413 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:01.422+0000 7ff868d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:01.415 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:01.422+0000 7ff868d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:01.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T07:03:01.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:03:01.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T07:03:01.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:03:01.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:03:01.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:01.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:03:01.690 INFO:tasks.workunit.client.0.smithi123.stderr:0 2024-03-29T07:03:01.690 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:01.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:01.951 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:01.958+0000 7ff868d297c0 -1 Falling back to public interface 2024-03-29T07:03:02.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:03.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:03.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:03.005 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:03:03.005 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:03:03.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:03.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:03.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:04.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:04.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:04.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:03:04.326 INFO:tasks.workunit.client.0.smithi123.stderr:2 2024-03-29T07:03:04.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:04.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:04.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:04.653 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:04.662+0000 7ff868d297c0 -1 osd.0 25 log_to_monitors true 2024-03-29T07:03:05.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:05.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:05.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:03:05.649 INFO:tasks.workunit.client.0.smithi123.stderr:3 2024-03-29T07:03:05.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:05.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:06.040 INFO:tasks.workunit.client.0.smithi123.stderr:osd.0 up in weight 1 up_from 29 up_thru 21 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/2587307236,v1:127.0.0.1:6803/2587307236] [v2:127.0.0.1:6804/2587307236,v1:127.0.0.1:6805/2587307236] exists,up 463465d0-7d60-46a6-9177-f117ac627042 2024-03-29T07:03:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:03:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:03:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:03:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-29T07:03:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:03:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:03:06.041 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:03:06.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:03:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:03:06.042 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:03:06.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:03:06.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:03:06.042 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:03:06.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:03:06.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:03:06.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:03:06.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:03:06.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:03:06.155 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:03:06.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:03:06.637 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:03:06.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:03:06.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:06.637 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:03:06.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-29T07:03:06.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-29T07:03:06.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-29T07:03:06.879 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:06.880 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:03:07.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-29T07:03:07.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-29T07:03:07.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672976' 2024-03-29T07:03:07.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:07.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-29T07:03:07.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:07.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:03:07.303 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-29T07:03:07.303 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:07.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-29T07:03:07.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-29T07:03:07.304 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.0 seq 124554051586 2024-03-29T07:03:07.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:03:07.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-29T07:03:07.652 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:03:08.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:03:08.654 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:03:09.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-29T07:03:09.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:09.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-29T07:03:09.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:09.017 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:03:09.018 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:09.018 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-29T07:03:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-29T07:03:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-29T07:03:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:waiting osd.1 seq 42949672976 2024-03-29T07:03:09.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:03:09.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-29T07:03:09.304 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:03:09.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:03:09.305 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:03:09.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-29T07:03:09.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:03:09.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:03:09.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:03:09.662 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:03:09.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:03:09.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:03:09.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:03:10.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T07:03:10.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:03:10.002 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:03:10.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:03:10.390 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-03-29T07:03:10.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-29T07:03:10.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-29T07:03:10.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-29T07:03:10.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T07:03:10.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-29T07:03:10.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T07:03:10.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:03:10.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:03:10.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:03:10.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:03:10.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:03:10.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:03:10.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:03:10.827 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-29T07:03:10.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:03:10.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:03:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-03-29T07:03:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-29T07:03:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-29T07:03:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-29T07:03:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-29T07:03:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-29T07:03:11.196 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-29T07:03:11.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-29T07:03:11.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-29T07:03:11.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-29T07:03:11.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-29T07:03:11.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-29T07:03:11.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-29T07:03:11.616 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-29T07:03:11.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-29T07:03:11.817 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:03:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:03:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:03:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:03:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:03:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:03:11.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:03:12.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-29T07:03:12.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:03:12.133 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:03:12.134 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:03:12.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-29T07:03:12.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:03:12.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:03:12.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-03-29T07:03:12.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-29T07:03:12.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-03-29T07:03:12.587 INFO:tasks.workunit.client.0.smithi123.stderr:No scrub information available for pg 2.0 2024-03-29T07:03:12.587 INFO:tasks.workunit.client.0.smithi123.stderr:error 2: (2) No such file or directory 2024-03-29T07:03:12.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-03-29T07:03:12.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-03-29T07:03:12.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-03-29T07:03:12.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-29T07:03:12.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:12.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T07:03:12.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:12.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-29T07:03:12.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-29T07:02:49.091446+0000 2024-03-29T07:03:12.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-03-29T07:03:13.182 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-03-29T07:03:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-03-29T07:02:49.091446+0000 last_deep_scrub_stamp 2024-03-29T07:03:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T07:03:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T07:02:49.091446+0000 2024-03-29T07:03:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-29T07:03:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T07:03:13.200 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:13.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-29T07:03:13.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:13.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T07:03:13.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:13.201 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-29T07:03:13.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:02:49.091446+0000 '>' 2024-03-29T07:02:49.091446+0000 2024-03-29T07:03:13.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:14.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:14.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:14.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-29T07:03:14.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:14.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T07:03:14.502 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:14.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-29T07:03:14.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:02:49.091446+0000 '>' 2024-03-29T07:02:49.091446+0000 2024-03-29T07:03:14.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:15.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:15.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:15.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-29T07:03:15.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:15.826 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T07:03:15.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:15.827 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-29T07:03:16.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:02:49.091446+0000 '>' 2024-03-29T07:02:49.091446+0000 2024-03-29T07:03:16.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:17.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:17.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:17.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-29T07:03:17.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:17.129 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-29T07:03:17.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:17.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-29T07:03:17.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:13.606016+0000 '>' 2024-03-29T07:02:49.091446+0000 2024-03-29T07:03:17.448 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T07:03:17.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-29T07:03:17.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-03-29T07:03:17.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-29T07:03:17.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-29T07:03:17.483 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:03:17.483 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:03:17.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:03:17.780 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:03:17.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:03:17.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:17.780 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:03:18.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-29T07:03:18.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-29T07:03:18.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-03-29T07:03:18.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:18.175 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:03:18.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-29T07:03:18.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-29T07:03:18.316 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-42949672979' 2024-03-29T07:03:18.317 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:18.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-03-29T07:03:18.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:18.318 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:03:18.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-03-29T07:03:18.319 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:18.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-29T07:03:18.320 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-03-29T07:03:18.320 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 124554051589 2024-03-29T07:03:18.321 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:03:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-03-29T07:03:18.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:03:19.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:03:19.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:03:19.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-03-29T07:03:19.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:19.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-29T07:03:19.956 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:19.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:03:19.958 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-29T07:03:19.958 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:19.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-29T07:03:19.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-29T07:03:19.960 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672979 2024-03-29T07:03:19.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:03:20.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-03-29T07:03:20.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-03-29T07:03:20.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:20.290 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:20.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:20.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:20.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:20.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-29T07:03:20.706 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:03:20.706 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": false, 2024-03-29T07:03:20.706 INFO:tasks.workunit.client.0.smithi123.stdout: "must": false, 2024-03-29T07:03:20.707 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "2024-03-22T07:01:40.716494+0000" 2024-03-29T07:03:20.707 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:20.721 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:20.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:20.722 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:21.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:13.606016+0000 '>' 2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:21.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:22.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:22.002 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:22.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:22.003 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:22.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:22.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:22.004 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:22.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:13.606016+0000 '>' 2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:22.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:23.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:23.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:23.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:23.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:23.306 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:23.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:23.307 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:23.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:13.606016+0000 '>' 2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:23.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:24.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:24.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:24.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:24.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:24.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:24.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:24.652 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:24.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:13.606016+0000 '>' 2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:24.957 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:25.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:25.960 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:25.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:25.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:25.960 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:25.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:25.961 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:26.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:13.606016+0000 '>' 2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:26.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:27.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:27.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:27.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:27.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:27.272 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:27.273 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:27.273 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:27.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:21.654612+0000 '>' 2024-03-29T07:03:13.606016+0000 2024-03-29T07:03:27.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T07:03:27.589 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-03-29T07:03:27.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-29T07:03:27.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-03-29T07:03:27.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-29T07:03:27.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-03-29T07:03:28.031 INFO:tasks.workunit.client.0.smithi123.stderr:nodeep-scrub is set 2024-03-29T07:03:28.048 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-03-29T07:03:28.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-29T07:03:28.208 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:03:28.208 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:03:28.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:03:28.501 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:03:28.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:03:28.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:28.501 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:03:28.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051592 2024-03-29T07:03:28.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051592 2024-03-29T07:03:28.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051592' 2024-03-29T07:03:28.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:28.639 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:03:28.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-29T07:03:28.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-29T07:03:28.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051592 1-42949672982' 2024-03-29T07:03:28.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:28.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051592 2024-03-29T07:03:28.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:28.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:03:28.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051592 2024-03-29T07:03:28.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:28.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051592 2024-03-29T07:03:28.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2024-03-29T07:03:28.781 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 124554051592 2024-03-29T07:03:28.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:03:29.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051592 2024-03-29T07:03:29.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:29.091 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-29T07:03:29.092 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:29.094 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:03:29.094 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-29T07:03:29.095 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-29T07:03:29.096 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-29T07:03:29.096 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672982 2024-03-29T07:03:29.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:03:29.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-03-29T07:03:29.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-03-29T07:03:34.412 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-29T07:03:34.555 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:03:34.555 INFO:tasks.workunit.client.0.smithi123.stdout: "deep": false, 2024-03-29T07:03:34.555 INFO:tasks.workunit.client.0.smithi123.stdout: "must": false, 2024-03-29T07:03:34.555 INFO:tasks.workunit.client.0.smithi123.stdout: "stamp": "2024-03-22T07:01:54.564861+0000" 2024-03-29T07:03:34.555 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:03:34.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-03-29T07:03:34.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-03-29T07:03:34.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-03-29T07:03:34.571 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-03-29T07:03:35.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-03-29T07:03:35.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-03-29T07:03:35.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-03-29T07:03:35.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-03-29T07:03:35.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-29T07:03:35.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-03-29T07:03:35.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-29T07:03:35.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-29T07:03:35.619 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:03:35.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:03:35.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:03:35.912 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:03:35.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:03:35.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:35.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:03:36.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051595 2024-03-29T07:03:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051595 2024-03-29T07:03:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051595' 2024-03-29T07:03:36.099 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:03:36.099 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:03:36.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-03-29T07:03:36.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-03-29T07:03:36.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051595 1-42949672985' 2024-03-29T07:03:36.238 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:36.239 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051595 2024-03-29T07:03:36.239 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:36.240 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:03:36.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051595 2024-03-29T07:03:36.241 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:36.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051595 2024-03-29T07:03:36.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051595' 2024-03-29T07:03:36.242 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 124554051595 2024-03-29T07:03:36.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:03:36.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051593 -lt 124554051595 2024-03-29T07:03:36.538 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:03:37.539 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:03:37.540 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:03:37.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051595 -lt 124554051595 2024-03-29T07:03:37.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:03:37.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-03-29T07:03:37.860 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:03:37.862 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:03:37.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-03-29T07:03:37.863 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:03:37.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-03-29T07:03:37.865 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672985 2024-03-29T07:03:37.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-03-29T07:03:37.866 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:03:38.215 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-03-29T07:03:38.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-03-29T07:03:38.216 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-03-29T07:03:38.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-03-29T07:03:38.216 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:38.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:38.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:38.217 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:38.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-03-22T07:01:54.564861+0000 2024-03-29T07:03:38.513 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-03-29T07:03:38.778 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 2.0 on osd.0 to scrub 2024-03-29T07:03:38.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-03-22T07:01:54.564861+0000 2024-03-29T07:03:38.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-29T07:03:38.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-22T07:01:54.564861+0000 2024-03-29T07:03:38.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T07:03:38.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T07:03:38.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:38.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:38.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:38.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:38.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:38.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:39.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-22T07:01:54.564861+0000 '>' 2024-03-22T07:01:54.564861+0000 2024-03-29T07:03:39.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:40.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:40.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:40.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:40.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:40.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:40.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:40.097 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:40.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-22T07:01:54.564861+0000 '>' 2024-03-22T07:01:54.564861+0000 2024-03-29T07:03:40.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:03:41.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:03:41.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:03:41.414 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-29T07:03:41.414 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-29T07:03:41.414 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:03:41.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:03:41.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-29T07:03:41.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:03:39.586645+0000 '>' 2024-03-22T07:01:54.564861+0000 2024-03-29T07:03:41.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T07:03:41.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-03-29T07:03:41.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-29T07:03:41.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-03-29T07:03:41.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-03-29T07:03:41.765 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T07:03:41.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T07:03:41.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T07:03:41.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T07:03:41.766 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:03:41.767 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:03:41.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:03:41.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:03:41.767 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:03:41.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:03:41.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T07:03:41.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:03:41.902 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:03:41.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:03:41.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T07:03:41.903 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T07:03:41.903 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T07:03:41.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T07:03:41.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T07:03:41.904 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T07:03:41.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:03:41.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T07:03:41.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T07:03:41.906 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:03:41.906 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:03:41.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T07:03:41.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T07:03:41.909 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T07:03:41.909 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:41.909 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:41.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T07:03:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T07:03:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T07:03:41.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T07:03:41.911 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T07:03:41.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:41.912 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:41.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T07:03:41.914 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T07:03:41.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T07:03:41.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T07:03:41.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T07:03:41.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T07:03:41.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T07:03:42.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T07:03:42.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:03:42.317 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:03:42.318 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:03:42.318 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:42.318 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:42.318 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:03:42.319 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T07:03:42.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T07:03:42.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T07:03:42.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T07:03:42.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T07:03:42.546 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T07:03:42.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T07:03:42.547 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T07:03:42.547 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T07:03:42.547 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T07:03:42.548 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T07:03:42.548 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:42.548 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:42.548 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T07:03:42.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T07:03:42.549 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T07:03:42.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T07:03:42.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T07:03:42.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T07:03:42.659 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T07:03:42.660 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T07:03:42.660 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T07:03:42.660 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T07:03:42.660 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T07:03:42.661 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T07:03:42.661 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:42.661 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:42.661 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T07:03:42.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T07:03:42.661 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T07:03:42.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-03-29T07:03:42.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T07:03:42.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T07:03:42.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T07:03:42.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T07:03:42.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T07:03:42.770 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T07:03:43.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T07:03:43.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:03:43.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:03:43.636 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:03:43.636 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:43.636 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:43.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:03:43.637 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T07:03:43.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T07:03:43.672 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:03:43.673 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-03-29T07:03:43.674 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-03-29T07:03:43.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T07:03:43.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:03:43.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T07:03:43.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T07:03:43.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T07:03:43.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T07:03:43.678 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T07:03:43.679 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:03:43.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:03:43.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:03:43.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T07:03:43.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:03:43.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T07:03:43.681 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T07:03:43.681 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9cd3a062-0295-4bfd-b17b-5fd5cbc0c558 2024-03-29T07:03:43.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9cd3a062-0295-4bfd-b17b-5fd5cbc0c558' 2024-03-29T07:03:43.682 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 9cd3a062-0295-4bfd-b17b-5fd5cbc0c558 2024-03-29T07:03:43.682 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T07:03:43.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPZwZmQIcdKhAAKiYYg8756kY9ap1Ltgj19A== 2024-03-29T07:03:43.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPZwZmQIcdKhAAKiYYg8756kY9ap1Ltgj19A=="}' 2024-03-29T07:03:43.697 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9cd3a062-0295-4bfd-b17b-5fd5cbc0c558 -i td/osd-scrub-repair/0/new.json 2024-03-29T07:03:43.964 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:03:43.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T07:03:43.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDPZwZmQIcdKhAAKiYYg8756kY9ap1Ltgj19A== --osd-uuid 9cd3a062-0295-4bfd-b17b-5fd5cbc0c558 2024-03-29T07:03:44.021 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:44.029+0000 7f9c3aad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:44.031 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:44.037+0000 7f9c3aad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:44.032 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:44.041+0000 7f9c3aad87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:44.033 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:44.041+0000 7f9c3aad87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:03:44.033 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:44.041+0000 7f9c3aad87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:03:44.033 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:44.041+0000 7f9c3aad87c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:03:44.123 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:44.129+0000 7f9c3aad87c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T07:03:50.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T07:03:50.475 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T07:03:50.476 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T07:03:50.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T07:03:50.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T07:03:50.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T07:03:50.869 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T07:03:50.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:03:50.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T07:03:50.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T07:03:50.871 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T07:03:50.904 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:50.908+0000 7f4620be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:50.928 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:50.932+0000 7f4620be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:50.930 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:50.936+0000 7f4620be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:03:51.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T07:03:51.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:03:51.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T07:03:51.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:03:51.190 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:03:51.191 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:03:51.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:51.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:03:51.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:51.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:51.490 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:51.496+0000 7f4620be97c0 -1 Falling back to public interface 2024-03-29T07:03:51.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:52.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:52.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:52.534 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:03:52.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:03:52.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:52.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:52.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:53.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:53.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:53.835 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:03:53.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:03:53.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:53.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:54.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:54.137 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:03:54.144+0000 7f4620be97c0 -1 osd.0 0 log_to_monitors true 2024-03-29T07:03:55.134 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:55.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:55.135 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:03:55.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:03:55.135 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:55.136 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:55.512 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:56.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:56.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:56.514 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T07:03:56.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T07:03:56.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:56.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:56.847 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:57.849 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:57.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:57.850 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T07:03:57.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T07:03:57.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:57.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:58.278 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:03:59.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:03:59.280 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:03:59.281 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T07:03:59.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T07:03:59.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:03:59.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:03:59.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:00.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:00.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:00.594 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T07:04:00.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T07:04:00.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:00.595 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:00.896 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:01.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:01.899 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:01.899 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T07:04:01.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T07:04:01.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:01.900 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:02.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:03.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:03.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:03.268 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T07:04:03.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T07:04:03.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:03.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:03.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:04.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:04.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:04.585 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T07:04:04.585 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T07:04:04.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:04.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:05.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:06.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:06.059 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:06.059 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T07:04:06.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T07:04:06.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:06.060 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:06.374 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:07.375 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:07.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:07.376 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T07:04:07.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T07:04:07.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:07.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:07.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:08.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:08.693 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:08.694 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T07:04:08.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T07:04:08.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:08.694 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:09.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:10.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:10.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:10.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T07:04:10.305 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T07:04:10.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:10.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:10.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:11.911 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T07:04:11.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T07:04:11.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:11.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:12.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:13.503 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:13.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:13.504 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T07:04:13.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T07:04:13.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:13.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:14.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:15.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:15.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:15.298 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T07:04:15.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T07:04:15.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:15.299 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:15.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:16.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:16.713 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:16.714 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T07:04:16.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T07:04:16.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:16.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:17.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:18.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:18.276 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:18.276 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T07:04:18.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T07:04:18.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:18.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:18.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:19.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:19.999 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:20.000 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T07:04:20.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T07:04:20.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:20.000 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:04:20.322 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1542073424,v1:127.0.0.1:6803/1542073424] [v2:127.0.0.1:6804/1542073424,v1:127.0.0.1:6805/1542073424] exists,up 9cd3a062-0295-4bfd-b17b-5fd5cbc0c558 2024-03-29T07:04:20.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:04:20.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:04:20.323 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T07:04:20.324 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:04:20.326 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T07:04:20.326 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:04:20.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T07:04:20.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:04:20.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T07:04:20.328 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T07:04:20.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38839e40-c209-4a43-a9ac-cf9d1a8135a5 2024-03-29T07:04:20.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 38839e40-c209-4a43-a9ac-cf9d1a8135a5' 2024-03-29T07:04:20.329 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 38839e40-c209-4a43-a9ac-cf9d1a8135a5 2024-03-29T07:04:20.329 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T07:04:20.350 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0ZwZmFttzFRAAvEEC/6Dci2DxZ95P0HGxBw== 2024-03-29T07:04:20.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0ZwZmFttzFRAAvEEC/6Dci2DxZ95P0HGxBw=="}' 2024-03-29T07:04:20.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38839e40-c209-4a43-a9ac-cf9d1a8135a5 -i td/osd-scrub-repair/1/new.json 2024-03-29T07:04:20.801 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:04:20.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T07:04:20.823 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD0ZwZmFttzFRAAvEEC/6Dci2DxZ95P0HGxBw== --osd-uuid 38839e40-c209-4a43-a9ac-cf9d1a8135a5 2024-03-29T07:04:20.859 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:20.864+0000 7fd318d177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:04:20.911 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:20.916+0000 7fd318d177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:04:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:20.920+0000 7fd318d177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:04:20.913 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:20.920+0000 7fd318d177c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:04:20.914 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:20.920+0000 7fd318d177c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:04:20.914 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:20.920+0000 7fd318d177c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:04:20.992 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:20.996+0000 7fd318d177c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T07:04:27.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T07:04:27.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T07:04:27.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T07:04:27.778 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T07:04:27.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T07:04:28.168 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T07:04:28.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T07:04:28.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:04:28.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T07:04:28.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T07:04:28.169 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T07:04:28.196 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:28.199+0000 7f1567f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:04:28.214 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:28.219+0000 7f1567f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:04:28.215 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:28.219+0000 7f1567f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:04:28.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T07:04:28.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:04:28.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T07:04:28.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:04:28.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:04:28.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:28.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:04:28.454 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:04:28.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:28.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:28.773 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:28.779+0000 7f1567f1f7c0 -1 Falling back to public interface 2024-03-29T07:04:28.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:29.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:29.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:29.781 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:04:29.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:04:29.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:29.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:30.089 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:31.091 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:04:31.091 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:04:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:31.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:31.397 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:31.519 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:04:31.523+0000 7f1567f1f7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T07:04:32.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:32.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:32.399 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:04:32.399 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:04:32.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:32.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:32.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:33.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:33.833 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:33.834 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T07:04:33.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T07:04:33.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:33.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:34.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:35.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:35.153 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:35.154 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T07:04:35.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T07:04:35.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:35.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:35.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:36.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:36.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:36.546 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T07:04:36.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T07:04:36.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:36.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:36.865 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:37.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:37.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:37.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T07:04:37.867 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T07:04:37.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:37.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:38.183 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:39.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:39.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:39.186 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T07:04:39.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T07:04:39.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:39.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:39.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:40.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:40.504 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:40.504 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T07:04:40.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T07:04:40.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:40.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:40.828 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:41.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:41.830 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:41.830 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T07:04:41.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T07:04:41.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:41.831 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:42.139 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:43.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:43.141 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:43.141 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T07:04:43.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T07:04:43.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:43.142 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:43.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:44.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:44.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:44.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T07:04:44.490 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T07:04:44.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:44.490 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:44.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:45.867 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:45.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:45.868 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T07:04:45.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T07:04:45.868 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:45.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:46.171 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:47.173 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:47.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:47.174 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T07:04:47.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T07:04:47.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:47.175 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:47.754 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:48.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:48.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:48.756 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T07:04:48.756 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T07:04:48.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:48.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:49.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:50.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:50.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:50.226 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T07:04:50.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T07:04:50.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:50.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:51.121 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:52.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:52.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:52.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T07:04:52.123 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T07:04:52.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:52.123 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:52.932 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:53.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:53.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:53.936 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T07:04:53.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T07:04:53.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:53.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:54.440 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:55.441 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:55.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:55.442 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T07:04:55.442 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T07:04:55.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:55.443 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:56.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:57.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:57.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:57.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T07:04:57.056 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T07:04:57.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:57.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:57.391 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:04:58.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:04:58.393 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:04:58.394 INFO:tasks.workunit.client.0.smithi123.stdout:21 2024-03-29T07:04:58.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-29T07:04:58.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:04:58.394 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:04:58.707 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3819952777,v1:127.0.0.1:6811/3819952777] [v2:127.0.0.1:6812/3819952777,v1:127.0.0.1:6813/3819952777] exists,up 38839e40-c209-4a43-a9ac-cf9d1a8135a5 2024-03-29T07:04:58.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:04:58.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:04:58.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:04:58.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-03-29T07:04:58.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-29T07:04:59.129 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'testpool' created 2024-03-29T07:04:59.149 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T07:05:00.151 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-03-29T07:05:00.793 INFO:tasks.workunit.client.0.smithi123.stderr:set pool 1 size to 2 2024-03-29T07:05:00.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-03-29T07:05:00.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:05:00.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:05:00.814 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:05:00.815 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:05:00.815 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:05:00.816 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:05:00.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:05:00.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:05:00.816 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:05:00.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:05:00.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:05:00.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:05:00.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:05:00.942 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:05:00.942 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:05:01.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:05:01.363 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:05:01.363 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:05:01.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:05:01.364 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:05:01.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-29T07:05:01.514 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-29T07:05:01.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-29T07:05:01.515 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:05:01.515 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:05:01.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-29T07:05:01.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-29T07:05:01.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672966' 2024-03-29T07:05:01.653 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:05:01.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-29T07:05:01.653 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:05:01.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:05:01.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-29T07:05:01.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:05:01.657 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-29T07:05:01.657 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836500 2024-03-29T07:05:01.658 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-29T07:05:01.658 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:05:01.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-29T07:05:01.938 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:05:02.940 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:05:02.941 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:05:03.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-29T07:05:03.260 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:05:03.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-29T07:05:03.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:05:03.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:05:03.264 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-29T07:05:03.265 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:05:03.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-29T07:05:03.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-29T07:05:03.266 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672966 2024-03-29T07:05:03.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:05:03.603 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-03-29T07:05:03.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:05:03.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:05:03.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:05:03.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T07:05:03.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:05:03.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:05:03.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:05:03.971 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:05:03.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:05:03.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:05:03.972 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:05:04.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T07:05:04.298 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:05:04.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:05:04.299 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:05:04.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T07:05:04.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:05:04.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:05:04.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-03-29T07:05:04.655 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-03-29T07:05:04.656 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-03-29T07:05:04.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:04.656 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:04.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:04.805 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:04.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:04.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:05.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:05.115 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:05.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:05.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:05.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:05.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:05.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:05.583 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:05.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:05.742 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:05.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:05.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:06.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:06.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:06.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:06.229 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:06.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:06.379 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:06.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:06.528 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:06.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:06.696 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:06.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:06.863 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:07.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:07.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:07.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:07.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:07.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:07.450 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:07.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:07.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:07.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:07.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:07.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:07.933 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:08.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:08.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:08.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:08.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:08.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:08.461 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:08.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:08.612 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:08.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:08.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:08.971 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:09.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:09.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:09.289 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:09.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-29T07:05:09.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:05:09.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-03-29T07:05:09.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T07:05:09.600 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-29T07:05:09.601 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-29T07:05:09.601 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T07:05:09.601 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-29T07:05:09.601 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T07:05:09.601 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-29T07:05:09.925 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T07:05:09.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-29T07:05:09.926 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T07:05:10.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-03-29T07:05:10.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-03-29T07:05:10.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-29T07:05:10.266 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-29T07:05:10.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-29T07:05:10.267 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T07:05:10.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-03-29T07:05:10.623 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-03-29T07:05:10.624 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T07:05:10.624 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-29T07:05:10.624 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-29T07:05:10.624 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T07:05:10.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-03-29T07:05:10.991 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T07:05:10.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:05:10.992 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:05:10.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:05:10.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:05:10.992 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:05:11.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:05:11.302 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T07:05:11.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:05:11.302 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:05:11.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:05:11.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:05:11.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:05:12.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:05:12.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-03-29T07:05:12.620 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:12.621 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-03-29T07:05:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:12.623 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-29T07:05:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:12.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-29T07:05:13.916 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-29T07:05:14.449 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:14.450 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-03-29T07:05:14.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:14.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-29T07:05:14.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:14.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:14.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:14.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:14.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:14.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-29T07:05:15.872 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:104778fc:::obj2:head# 2024-03-29T07:05:16.406 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:16.407 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-03-29T07:05:16.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:16.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-29T07:05:16.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:16.409 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:16.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:16.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:16.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:16.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-29T07:05:17.382 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:8dd16f86:::obj3:head# 2024-03-29T07:05:17.918 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:17.919 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-03-29T07:05:17.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:17.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-29T07:05:17.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:17.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:17.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:17.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:17.921 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:17.922 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-29T07:05:19.011 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:0ee9ae15:::obj4:head# 2024-03-29T07:05:19.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:19.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:19.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-29T07:05:20.396 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:c52c9666:::obj5:head# 2024-03-29T07:05:20.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:20.935 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-03-29T07:05:20.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:20.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-29T07:05:20.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:20.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:20.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:20.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:20.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:20.937 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-29T07:05:21.740 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:233a42c1:::obj6:head# 2024-03-29T07:05:22.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:22.275 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:22.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-29T07:05:23.108 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:9a09113a:::obj7:head# 2024-03-29T07:05:23.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:23.643 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-03-29T07:05:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:23.645 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-29T07:05:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:23.646 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-29T07:05:24.460 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:a557efb1:::obj8:head# 2024-03-29T07:05:24.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:24.995 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:24.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-29T07:05:25.779 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:fe351e27:::obj9:head# 2024-03-29T07:05:26.322 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:26.323 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-03-29T07:05:26.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:26.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-29T07:05:26.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:26.325 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:26.326 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-29T07:05:27.206 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:d1337354:::obj10:head# 2024-03-29T07:05:27.740 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:27.741 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:27.743 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-29T07:05:28.508 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:eeae2a94:::obj11:head# 2024-03-29T07:05:29.294 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:29.295 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-03-29T07:05:29.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:29.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-29T07:05:29.297 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:29.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:29.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:29.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:29.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:29.298 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-29T07:05:30.078 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:af5f95cb:::obj12:head# 2024-03-29T07:05:30.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:30.871 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-03-29T07:05:30.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:30.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-29T07:05:30.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:30.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:30.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:30.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:30.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:30.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-29T07:05:31.623 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:a61ad63e:::obj13:head# 2024-03-29T07:05:32.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:32.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:32.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-29T07:05:32.966 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:e0a44829:::obj14:head# 2024-03-29T07:05:33.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:33.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:33.505 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-29T07:05:34.258 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:ab946124:::obj15:head# 2024-03-29T07:05:34.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:34.795 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-03-29T07:05:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-29T07:05:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:34.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:34.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:34.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-29T07:05:35.544 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:461f8b5e:::obj16:head# 2024-03-29T07:05:36.076 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:36.077 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:36.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-29T07:05:36.945 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:24fc6e92:::obj17:head# 2024-03-29T07:05:37.482 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:37.482 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-03-29T07:05:37.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:37.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-29T07:05:37.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:37.484 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:37.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:37.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:37.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:37.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-29T07:05:38.246 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:b9836a99:::obj18:head# 2024-03-29T07:05:38.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:38.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-03-29T07:05:38.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-29T07:05:38.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-29T07:05:38.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:38.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:38.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:05:38.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:38.793 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:38.794 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-29T07:05:39.541 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:00b31b6c:::obj19:head# 2024-03-29T07:05:40.078 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-29T07:05:40.079 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-03-29T07:05:40.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-29T07:05:40.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-29T07:05:40.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:05:40.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:05:40.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:05:40.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:05:40.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:40.082 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-29T07:05:40.891 INFO:tasks.workunit.client.0.smithi123.stdout:remove #1:24b5611b:::obj20:head# 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:05:41.427 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T07:05:41.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T07:05:41.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:05:41.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:05:41.428 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:05:41.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:05:41.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:05:41.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:05:41.430 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:05:41.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:05:41.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T07:05:41.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T07:05:41.431 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T07:05:41.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:05:41.432 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T07:05:41.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T07:05:41.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:05:41.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:05:41.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:05:41.467 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:41.474+0000 7f55406117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:05:41.469 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:41.478+0000 7f55406117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:05:41.471 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:41.478+0000 7f55406117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:05:41.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T07:05:41.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:05:41.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T07:05:41.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:05:41.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:05:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:41.747 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:05:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:05:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:05:41.747 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:42.006 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:42.014+0000 7f55406117c0 -1 Falling back to public interface 2024-03-29T07:05:42.079 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:05:43.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:05:43.080 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:43.080 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:05:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:05:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:43.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:05:43.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:05:44.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:05:44.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:44.400 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:05:44.400 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:05:44.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:44.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:05:44.596 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:44.606+0000 7f55406117c0 -1 osd.1 17 log_to_monitors true 2024-03-29T07:05:44.726 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:05:45.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:05:45.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:45.729 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:05:45.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:05:45.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:45.729 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:05:46.046 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/1975288882,v1:127.0.0.1:6803/1975288882] [v2:127.0.0.1:6804/1975288882,v1:127.0.0.1:6805/1975288882] exists,up 38839e40-c209-4a43-a9ac-cf9d1a8135a5 2024-03-29T07:05:46.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:05:46.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:05:46.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:05:46.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:05:46.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:05:46.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:05:46.053 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:05:46.053 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:05:46.053 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:05:46.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:05:46.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T07:05:46.054 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T07:05:46.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T07:05:46.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:05:46.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T07:05:46.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T07:05:46.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:05:46.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:05:46.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:05:46.085 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:46.094+0000 7fa9f9a9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:05:46.108 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:46.118+0000 7fa9f9a9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:05:46.109 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:46.118+0000 7fa9f9a9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:05:46.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T07:05:46.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:05:46.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T07:05:46.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:05:46.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:05:46.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:46.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:05:46.387 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:05:46.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:05:46.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:46.641 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:46.650+0000 7fa9f9a9d7c0 -1 Falling back to public interface 2024-03-29T07:05:46.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:05:47.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:05:47.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:47.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:05:47.724 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:05:47.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:47.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:05:48.040 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:05:49.042 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:05:49.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:49.043 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:05:49.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:05:49.043 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:49.044 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:05:49.330 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:05:49.338+0000 7fa9f9a9d7c0 -1 osd.0 15 log_to_monitors true 2024-03-29T07:05:49.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:05:50.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:05:50.357 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:05:50.357 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:05:50.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:05:50.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:05:50.358 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:05:50.675 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 16 last_clean_interval [5,15) [v2:127.0.0.1:6810/104892803,v1:127.0.0.1:6811/104892803] [v2:127.0.0.1:6812/104892803,v1:127.0.0.1:6813/104892803] exists,up 9cd3a062-0295-4bfd-b17b-5fd5cbc0c558 2024-03-29T07:05:50.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:05:50.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:05:50.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:05:50.676 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-03-29T07:05:50.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:05:50.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:05:50.677 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:05:50.677 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:05:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:05:50.678 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:05:50.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:05:50.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:05:50.678 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:05:50.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:05:50.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:05:50.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:05:50.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:05:50.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:05:50.778 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:05:51.092 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:05:51.092 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:05:51.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:05:51.093 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:05:51.093 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:05:51.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-03-29T07:05:51.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-03-29T07:05:51.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-03-29T07:05:51.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:05:51.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:05:51.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345923 2024-03-29T07:05:51.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345923 2024-03-29T07:05:51.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-85899345923' 2024-03-29T07:05:51.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:05:51.385 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-03-29T07:05:51.386 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:05:51.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:05:51.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-03-29T07:05:51.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:05:51.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-03-29T07:05:51.389 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-03-29T07:05:51.389 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 98784247810 2024-03-29T07:05:51.389 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:05:51.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-03-29T07:05:51.679 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:05:52.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:05:52.682 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:05:53.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-03-29T07:05:53.052 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:05:53.052 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345923 2024-03-29T07:05:53.053 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:05:53.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:05:53.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345923 2024-03-29T07:05:53.055 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:05:53.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345923 2024-03-29T07:05:53.057 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 85899345923 2024-03-29T07:05:53.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345923' 2024-03-29T07:05:53.057 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:05:53.398 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345923 2024-03-29T07:05:53.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:05:53.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:05:53.399 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:05:53.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T07:05:53.785 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:05:53.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:05:53.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:05:53.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:05:53.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:05:53.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:05:53.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:05:54.120 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T07:05:54.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:05:54.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:05:54.121 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:05:54.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T07:05:54.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:05:54.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:05:54.509 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-03-29T07:05:54.510 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-29T07:05:54.510 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-29T07:05:54.510 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:05:54.511 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:05:54.511 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:05:54.511 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:05:54.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T07:04:59.078540+0000 2024-03-29T07:05:54.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-29T07:05:55.127 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-29T07:05:55.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-29T07:04:59.078540+0000 2024-03-29T07:05:55.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-29T07:05:55.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T07:04:59.078540+0000 2024-03-29T07:05:55.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T07:05:55.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T07:05:55.143 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:05:55.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:05:55.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:05:55.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:05:55.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:05:55.144 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:05:55.424 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:04:59.078540+0000 '>' 2024-03-29T07:04:59.078540+0000 2024-03-29T07:05:55.425 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:05:56.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:05:56.426 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:05:56.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:05:56.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:05:56.427 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:05:56.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:05:56.428 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:05:56.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:04:59.078540+0000 '>' 2024-03-29T07:04:59.078540+0000 2024-03-29T07:05:56.744 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:05:57.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:05:57.746 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:05:57.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:05:57.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:05:57.747 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:05:57.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:05:57.748 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:05:58.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:04:59.078540+0000 '>' 2024-03-29T07:04:59.078540+0000 2024-03-29T07:05:58.066 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:05:59.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:05:59.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:05:59.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:05:59.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:05:59.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:05:59.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:05:59.070 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:05:59.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:04:59.078540+0000 '>' 2024-03-29T07:04:59.078540+0000 2024-03-29T07:05:59.385 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:06:00.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:06:00.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:06:00.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:06:00.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:06:00.387 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:06:00.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:06:00.388 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:06:00.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:05:55.577319+0000 '>' 2024-03-29T07:04:59.078540+0000 2024-03-29T07:06:00.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T07:06:00.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-03-29T07:06:00.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:06:00.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:06:00.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:06:00.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:06:00.705 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:06:00.705 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:06:00.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:06:00.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:06:00.705 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:06:00.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:06:00.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:06:00.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:06:00.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:06:00.822 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:06:00.823 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:06:01.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:06:01.122 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:06:01.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:06:01.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:06:01.123 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:06:01.290 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247813 2024-03-29T07:06:01.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247813 2024-03-29T07:06:01.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247813' 2024-03-29T07:06:01.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:06:01.291 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:06:01.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-03-29T07:06:01.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-03-29T07:06:01.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247813 1-85899345926' 2024-03-29T07:06:01.433 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:06:01.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247813 2024-03-29T07:06:01.433 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:06:01.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:06:01.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247813 2024-03-29T07:06:01.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:06:01.436 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247813 2024-03-29T07:06:01.436 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 98784247813 2024-03-29T07:06:01.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247813' 2024-03-29T07:06:01.437 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:06:01.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247812 -lt 98784247813 2024-03-29T07:06:01.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:06:02.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:06:02.730 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:06:03.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247813 2024-03-29T07:06:03.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:06:03.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345926 2024-03-29T07:06:03.051 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:06:03.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:06:03.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345926 2024-03-29T07:06:03.054 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:06:03.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-03-29T07:06:03.055 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345926' 2024-03-29T07:06:03.055 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 85899345926 2024-03-29T07:06:03.056 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:06:03.383 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345926 -lt 85899345926 2024-03-29T07:06:03.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:06:03.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:06:03.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:06:03.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T07:06:03.790 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:06:03.790 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:06:03.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:06:03.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:06:03.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:06:03.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:06:03.791 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:06:04.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T07:06:04.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:06:04.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:06:04.130 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:06:04.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T07:06:04.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:06:04.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:06:04.494 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-03-29T07:06:04.825 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T07:06:04.825 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-29T07:05:56.978493+0000 15'30 24:123 [1,0] 1 [1,0] 1 15'30 2024-03-29T07:05:55.577319+0000 15'30 2024-03-29T07:05:55.577319+0000 0 1 periodic scrub scheduled @ 2024-03-30T07:05:55.577319+0000 20 0 2024-03-29T07:06:04.825 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T07:06:04.825 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T07:06:04.826 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T07:06:04.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-03-29T07:06:04.841 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:06:04.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:06:05.155 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:06:05.155 INFO:tasks.workunit.client.0.smithi123.stderr:1' 2024-03-29T07:06:05.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:06:05.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:06:05.156 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:06:05.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247815 2024-03-29T07:06:05.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247815 2024-03-29T07:06:05.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247815' 2024-03-29T07:06:05.310 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:06:05.310 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:06:05.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-03-29T07:06:05.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-03-29T07:06:05.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247815 1-85899345928' 2024-03-29T07:06:05.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:06:05.453 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247815 2024-03-29T07:06:05.454 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:06:05.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:06:05.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247815 2024-03-29T07:06:05.455 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:06:05.456 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247815 2024-03-29T07:06:05.457 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 98784247815 2024-03-29T07:06:05.457 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247815' 2024-03-29T07:06:05.457 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:06:05.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247815 2024-03-29T07:06:05.751 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:06:06.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:06:06.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:06:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247815 -lt 98784247815 2024-03-29T07:06:07.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:06:07.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:06:07.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345928 2024-03-29T07:06:07.067 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:06:07.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345928 2024-03-29T07:06:07.069 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:06:07.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-03-29T07:06:07.070 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345928' 2024-03-29T07:06:07.071 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 85899345928 2024-03-29T07:06:07.071 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:06:07.431 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345928 2024-03-29T07:06:07.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-03-29T07:06:07.432 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_bytes": 210, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects": 30, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_object_clones": 0, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_object_copies": 60, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_missing_on_primary": 0, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_missing": 0, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_degraded": 0, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_misplaced": 0, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_unfound": 0, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_dirty": 30, 2024-03-29T07:06:07.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_whiteouts": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_read": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_read_kb": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_write": 30, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_write_kb": 30, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_scrub_errors": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_shallow_scrub_errors": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_deep_scrub_errors": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_recovered": 20, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_bytes_recovered": 140, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_keys_recovered": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_omap": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_hit_set_archive": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_bytes_hit_set_archive": 0, 2024-03-29T07:06:07.625 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush_kb": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict_kb": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_promote": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush_mode_high": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush_mode_low": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict_mode_some": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict_mode_full": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_pinned": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_legacy_snapsets": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_large_omap_objects": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_manifest": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_omap_bytes": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_omap_keys": 0, 2024-03-29T07:06:07.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_repaired": 20 2024-03-29T07:06:07.627 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:06:07.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-03-29T07:06:07.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-03-29T07:06:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-03-29T07:06:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-03-29T07:06:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-29T07:06:07.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-03-29T07:06:08.028 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:06:08.043 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "osd": 1, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "up_from": 20, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "seq": 85899345928, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "num_pgs": 1, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "num_osds": 1, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_osds": 1, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_omap_osds": 1, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "kb": 104857600, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used": 28552, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_data": 320, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_omap": 4, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_meta": 28219, 2024-03-29T07:06:08.044 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_avail": 104829048, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "statfs": { 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "total": 107374182400, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "available": 107344945152, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "internally_reserved": 0, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "allocated": 327680, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "data_stored": 53284, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed": 0, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_allocated": 0, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_original": 0, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "omap_allocated": 4398, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "internal_metadata": 28896978 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:08.045 INFO:tasks.workunit.client.0.smithi123.stdout: "hb_peers": [ 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: 0 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: "snap_trim_queue_len": 0, 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: "num_snap_trimming": 0, 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: "num_shards_repaired": 10, 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: "op_queue_age_hist": { 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: "histogram": [], 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: "upper_bound": 1 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:08.046 INFO:tasks.workunit.client.0.smithi123.stdout: "perf_stat": { 2024-03-29T07:06:08.047 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ms": 0, 2024-03-29T07:06:08.047 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ms": 0, 2024-03-29T07:06:08.047 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ns": 0, 2024-03-29T07:06:08.047 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ns": 0 2024-03-29T07:06:08.047 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:08.047 INFO:tasks.workunit.client.0.smithi123.stdout: "alerts": [] 2024-03-29T07:06:08.047 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:06:08.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-29T07:06:08.048 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-03-29T07:06:08.317 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:06:08.333 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-03-29T07:06:08.333 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-03-29T07:06:08.334 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-03-29T07:06:08.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-29T07:06:08.335 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-29T07:06:08.609 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "osd": 0, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "up_from": 23, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "seq": 98784247815, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_pgs": 1, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_osds": 1, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_osds": 1, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_omap_osds": 1, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "kb": 104857600, 2024-03-29T07:06:08.624 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used": 28552, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_data": 320, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_omap": 4, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_meta": 28219, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_avail": 104829048, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "statfs": { 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "total": 107374182400, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "available": 107344945152, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "internally_reserved": 0, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "allocated": 327680, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "data_stored": 53284, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed": 0, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_allocated": 0, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_original": 0, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "omap_allocated": 4401, 2024-03-29T07:06:08.625 INFO:tasks.workunit.client.0.smithi123.stdout: "internal_metadata": 28896975 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "hb_peers": [ 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: 1 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "snap_trim_queue_len": 0, 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_snap_trimming": 0, 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "num_shards_repaired": 10, 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "op_queue_age_hist": { 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "histogram": [], 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "upper_bound": 1 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "perf_stat": { 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ms": 0, 2024-03-29T07:06:08.626 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ms": 0, 2024-03-29T07:06:08.627 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ns": 0, 2024-03-29T07:06:08.627 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ns": 0 2024-03-29T07:06:08.627 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:08.627 INFO:tasks.workunit.client.0.smithi123.stdout: "alerts": [] 2024-03-29T07:06:08.627 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:06:08.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-29T07:06:08.627 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-29T07:06:08.892 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:06:08.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-03-29T07:06:08.909 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-03-29T07:06:08.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-03-29T07:06:08.910 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-29T07:06:08.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-03-29T07:06:09.184 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout: "up_from": 0, 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout: "seq": 0, 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout: "num_pgs": 2, 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout: "num_osds": 2, 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_osds": 2, 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_omap_osds": 2, 2024-03-29T07:06:09.199 INFO:tasks.workunit.client.0.smithi123.stdout: "kb": 209715200, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used": 57104, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_data": 640, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_omap": 8, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_meta": 56439, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_avail": 209658096, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "statfs": { 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "total": 214748364800, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "available": 214689890304, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "internally_reserved": 0, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "allocated": 655360, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "data_stored": 106568, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed": 0, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_allocated": 0, 2024-03-29T07:06:09.200 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_original": 0, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "omap_allocated": 8799, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "internal_metadata": 57793953 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "hb_peers": [], 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "snap_trim_queue_len": 0, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "num_snap_trimming": 0, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "num_shards_repaired": 20, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "op_queue_age_hist": { 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "histogram": [], 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "upper_bound": 1 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "perf_stat": { 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ms": 0, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ms": 0, 2024-03-29T07:06:09.201 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ns": 0, 2024-03-29T07:06:09.202 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ns": 0 2024-03-29T07:06:09.202 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:06:09.202 INFO:tasks.workunit.client.0.smithi123.stdout: "alerts": [], 2024-03-29T07:06:09.202 INFO:tasks.workunit.client.0.smithi123.stdout: "network_ping_times": [] 2024-03-29T07:06:09.202 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:06:09.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-29T07:06:09.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-03-29T07:06:09.475 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:06:09.491 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-03-29T07:06:09.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-03-29T07:06:09.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-29T07:06:09.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T07:06:09.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T07:06:09.492 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T07:06:09.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:06:09.493 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:06:09.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:06:09.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:06:09.493 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:06:09.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:06:09.604 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T07:06:09.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T07:06:09.605 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T07:06:09.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T07:06:09.606 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T07:06:09.607 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T07:06:09.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:06:09.608 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T07:06:09.609 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T07:06:09.609 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:06:09.610 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:06:09.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T07:06:09.611 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T07:06:09.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T07:06:09.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:09.628 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:09.628 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-29T07:06:09.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T07:06:09.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:06:09.631 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:06:09.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:06:09.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:06:09.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:06:09.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:06:09.633 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T07:06:09.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T07:06:09.635 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T07:06:09.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T07:06:09.636 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T07:06:09.636 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T07:06:09.637 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:06:09.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T07:06:09.638 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T07:06:09.638 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:06:09.639 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:06:09.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-29T07:06:09.640 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T07:06:09.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T07:06:09.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:09.642 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:09.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T07:06:09.642 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T07:06:09.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T07:06:09.643 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-29T07:06:09.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-29T07:06:09.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:09.644 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:09.644 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.61544 2024-03-29T07:06:09.647 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-29T07:06:09.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-29T07:06:09.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-29T07:06:09.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-29T07:06:09.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-03-29T07:06:09.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-03-29T07:06:09.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-29T07:06:09.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-29T07:06:10.029 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-29T07:06:10.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:06:10.030 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:06:10.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:06:10.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:10.030 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:10.031 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:06:10.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-29T07:06:10.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-29T07:06:10.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-29T07:06:10.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T07:06:10.244 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T07:06:10.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-29T07:06:10.245 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-29T07:06:10.246 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T07:06:10.246 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T07:06:10.246 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T07:06:10.246 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T07:06:10.246 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:10.246 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:10.246 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T07:06:10.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T07:06:10.247 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get fsid 2024-03-29T07:06:10.366 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-29T07:06:10.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-29T07:06:10.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-29T07:06:10.367 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-29T07:06:10.368 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-29T07:06:10.368 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-29T07:06:10.368 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-29T07:06:10.368 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-29T07:06:10.368 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-29T07:06:10.368 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:10.369 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.61544/ceph-mon.a.asok 2024-03-29T07:06:10.369 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-29T07:06:10.369 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.61544/ceph-mon.a.asok config get mon_host 2024-03-29T07:06:10.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-03-29T07:06:10.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-29T07:06:10.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-29T07:06:10.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-29T07:06:10.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-29T07:06:10.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-29T07:06:10.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-29T07:06:11.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-29T07:06:11.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:06:11.357 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:06:11.358 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:06:11.358 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:11.358 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:11.358 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:06:11.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T07:06:11.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-29T07:06:11.396 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:06:11.397 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-03-29T07:06:11.398 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:06:11.401 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:11.402 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:06:11.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:06:11.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:06:11.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T07:06:11.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T07:06:11.404 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9312f8f-1cb2-4093-845c-e6964818107a 2024-03-29T07:06:11.405 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d9312f8f-1cb2-4093-845c-e6964818107a' 2024-03-29T07:06:11.406 INFO:tasks.workunit.client.0.smithi123.stdout:add osd0 d9312f8f-1cb2-4093-845c-e6964818107a 2024-03-29T07:06:11.406 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T07:06:11.421 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjaAZmbiG2GRAA6WLvBctEig0QAHX0uQFxzQ== 2024-03-29T07:06:11.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjaAZmbiG2GRAA6WLvBctEig0QAHX0uQFxzQ=="}' 2024-03-29T07:06:11.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9312f8f-1cb2-4093-845c-e6964818107a -i td/osd-scrub-repair/0/new.json 2024-03-29T07:06:11.669 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:06:11.685 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-29T07:06:11.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBjaAZmbiG2GRAA6WLvBctEig0QAHX0uQFxzQ== --osd-uuid d9312f8f-1cb2-4093-845c-e6964818107a 2024-03-29T07:06:11.722 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:11.729+0000 7f70867b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:11.724 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:11.733+0000 7f70867b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:11.725 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:11.733+0000 7f70867b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:11.726 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:11.733+0000 7f70867b77c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:06:11.726 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:11.733+0000 7f70867b77c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:06:11.726 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:11.733+0000 7f70867b77c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-29T07:06:11.828 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:11.833+0000 7f70867b77c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-29T07:06:18.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-29T07:06:18.237 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T07:06:18.239 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd0 key to auth repository 2024-03-29T07:06:18.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-29T07:06:18.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T07:06:18.624 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T07:06:18.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-29T07:06:18.624 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:06:18.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T07:06:18.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T07:06:18.625 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T07:06:18.656 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:18.665+0000 7f18955fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:18.659 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:18.665+0000 7f18955fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:18.660 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:18.669+0000 7f18955fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:18.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-29T07:06:18.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:06:18.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T07:06:18.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:06:18.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:06:18.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:18.965 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:06:18.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:06:18.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:18.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:19.226 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:19.233+0000 7f18955fe7c0 -1 Falling back to public interface 2024-03-29T07:06:19.312 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:20.313 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:20.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:20.314 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:06:20.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:06:20.314 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:20.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:20.629 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:21.630 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:21.647 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:21.647 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:06:21.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:06:21.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:21.648 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:21.909 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:21.917+0000 7f18955fe7c0 -1 osd.0 0 log_to_monitors true 2024-03-29T07:06:21.956 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:22.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:22.958 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:22.958 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:06:22.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:06:22.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:22.959 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:23.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:24.272 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:24.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:24.273 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T07:06:24.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T07:06:24.273 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:24.274 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:24.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:25.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:25.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:25.577 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T07:06:25.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T07:06:25.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:25.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:25.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:26.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:26.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:26.914 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T07:06:26.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T07:06:26.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:26.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:27.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:28.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:28.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:28.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T07:06:28.228 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T07:06:28.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:28.228 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:28.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:29.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:29.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:29.558 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T07:06:29.558 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T07:06:29.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:29.559 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:29.946 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:30.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:30.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:30.948 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T07:06:30.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T07:06:30.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:30.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:31.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:32.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:32.268 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:32.269 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T07:06:32.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T07:06:32.269 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:32.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:32.590 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:33.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:33.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:33.592 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T07:06:33.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T07:06:33.592 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:33.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:33.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:34.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:34.972 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:34.972 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T07:06:34.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T07:06:34.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:34.973 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:35.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:36.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:36.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:36.295 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T07:06:36.295 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T07:06:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:36.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:36.948 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:37.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:37.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:37.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T07:06:37.950 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T07:06:37.950 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:37.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:38.780 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:39.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:39.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:39.782 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T07:06:39.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T07:06:39.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:39.783 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:40.722 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:41.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:41.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:41.724 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T07:06:41.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T07:06:41.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:41.725 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:42.586 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:43.587 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:43.588 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T07:06:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T07:06:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:43.588 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:44.174 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:45.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:45.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:45.176 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T07:06:45.177 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T07:06:45.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:45.177 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:45.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:46.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:46.500 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:46.500 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T07:06:46.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T07:06:46.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:46.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:47.336 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:48.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:48.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:48.339 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T07:06:48.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T07:06:48.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:48.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:06:48.702 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2499626711,v1:127.0.0.1:6803/2499626711] [v2:127.0.0.1:6804/2499626711,v1:127.0.0.1:6805/2499626711] exists,up d9312f8f-1cb2-4093-845c-e6964818107a 2024-03-29T07:06:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:06:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:06:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:06:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T07:06:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:06:48.704 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T07:06:48.705 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T07:06:48.706 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:06:48.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T07:06:48.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:06:48.706 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:06:48.706 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:06:48.706 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:06:48.706 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:06:48.708 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:06:48.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:06:48.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:06:48.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T07:06:48.708 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=85ce611f-a6e8-4fdf-96cf-f4cc812e57b4 2024-03-29T07:06:48.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 85ce611f-a6e8-4fdf-96cf-f4cc812e57b4' 2024-03-29T07:06:48.710 INFO:tasks.workunit.client.0.smithi123.stdout:add osd1 85ce611f-a6e8-4fdf-96cf-f4cc812e57b4 2024-03-29T07:06:48.710 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T07:06:48.727 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIaAZmI3z3KxAA40d+b55bSB25zHAOGcHtUA== 2024-03-29T07:06:48.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIaAZmI3z3KxAA40d+b55bSB25zHAOGcHtUA=="}' 2024-03-29T07:06:48.728 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 85ce611f-a6e8-4fdf-96cf-f4cc812e57b4 -i td/osd-scrub-repair/1/new.json 2024-03-29T07:06:49.083 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:06:49.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-29T07:06:49.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCIaAZmI3z3KxAA40d+b55bSB25zHAOGcHtUA== --osd-uuid 85ce611f-a6e8-4fdf-96cf-f4cc812e57b4 2024-03-29T07:06:49.138 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:49.144+0000 7fb2f4ab57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:49.176 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:49.184+0000 7fb2f4ab57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:49.177 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:49.184+0000 7fb2f4ab57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:49.178 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:49.184+0000 7fb2f4ab57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:06:49.178 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:49.184+0000 7fb2f4ab57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:06:49.178 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:49.184+0000 7fb2f4ab57c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-29T07:06:49.252 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:49.260+0000 7fb2f4ab57c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-29T07:06:56.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-29T07:06:56.045 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T07:06:56.047 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd1 key to auth repository 2024-03-29T07:06:56.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-29T07:06:56.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T07:06:56.438 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T07:06:56.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-29T07:06:56.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T07:06:56.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T07:06:56.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T07:06:56.439 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:06:56.471 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:56.476+0000 7f075e3ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:56.496 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:56.504+0000 7f075e3ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:56.497 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:56.504+0000 7f075e3ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:06:56.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-29T07:06:56.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:06:56.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T07:06:56.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:06:56.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:06:56.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:56.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:06:56.777 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:06:56.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:56.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:06:57.054 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:57.060+0000 7f075e3ee7c0 -1 Falling back to public interface 2024-03-29T07:06:57.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:58.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:58.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:58.097 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:06:58.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:06:58.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:58.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:06:58.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:59.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:06:59.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:06:59.415 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:06:59.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:06:59.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:06:59.416 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:06:59.721 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:06:59.738 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:06:59.744+0000 7f075e3ee7c0 -1 osd.1 0 log_to_monitors true 2024-03-29T07:07:00.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:00.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:00.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:07:00.724 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:07:00.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:00.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:01.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:02.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:02.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:02.036 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T07:07:02.036 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T07:07:02.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:02.037 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:02.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:03.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:03.348 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:03.348 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T07:07:03.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T07:07:03.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:03.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:03.709 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:04.710 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:04.711 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:04.711 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T07:07:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T07:07:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:04.712 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:05.032 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:06.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:06.034 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:06.034 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T07:07:06.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T07:07:06.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:06.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:06.349 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:07.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:07.351 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:07.352 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T07:07:07.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T07:07:07.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:07.352 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:07.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:08.810 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:08.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:08.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T07:07:08.811 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T07:07:08.811 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:08.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:09.130 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:10.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:10.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:10.132 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T07:07:10.132 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T07:07:10.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:10.133 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:10.452 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:11.453 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:11.454 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T07:07:11.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T07:07:11.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:11.454 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:11.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:12.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:12.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:12.776 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T07:07:12.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T07:07:12.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:12.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:13.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:14.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:14.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:14.097 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T07:07:14.098 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T07:07:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:14.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:14.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:15.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:15.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:15.414 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T07:07:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T07:07:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:15.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:16.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:17.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:17.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:17.158 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T07:07:17.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T07:07:17.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:17.159 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:17.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:18.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:18.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:18.552 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T07:07:18.553 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T07:07:18.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:18.554 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:19.127 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:20.128 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:20.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:20.129 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T07:07:20.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T07:07:20.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:20.129 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:21.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:22.006 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:22.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:22.007 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T07:07:22.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T07:07:22.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:22.007 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:22.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:23.844 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:23.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:23.845 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T07:07:23.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T07:07:23.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:23.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:24.460 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:25.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:25.462 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:25.463 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T07:07:25.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T07:07:25.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:25.463 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:07:25.774 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3713942099,v1:127.0.0.1:6811/3713942099] [v2:127.0.0.1:6812/3713942099,v1:127.0.0.1:6813/3713942099] exists,up 85ce611f-a6e8-4fdf-96cf-f4cc812e57b4 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-29T07:07:25.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-29T07:07:25.778 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-29T07:07:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:07:25.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-29T07:07:25.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:07:25.779 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:07:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:07:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:07:25.779 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:07:25.781 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:07:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-29T07:07:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:07:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-29T07:07:25.782 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-29T07:07:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a90bce8c-2f3b-47d6-8790-977145d7a877 2024-03-29T07:07:25.782 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a90bce8c-2f3b-47d6-8790-977145d7a877' 2024-03-29T07:07:25.783 INFO:tasks.workunit.client.0.smithi123.stdout:add osd2 a90bce8c-2f3b-47d6-8790-977145d7a877 2024-03-29T07:07:25.783 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-29T07:07:25.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtaAZmtdV6MBAAzew80oARlcFQNODFOirvzQ== 2024-03-29T07:07:25.803 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtaAZmtdV6MBAAzew80oARlcFQNODFOirvzQ=="}' 2024-03-29T07:07:25.804 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a90bce8c-2f3b-47d6-8790-977145d7a877 -i td/osd-scrub-repair/2/new.json 2024-03-29T07:07:26.269 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:07:26.291 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-29T07:07:26.293 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCtaAZmtdV6MBAAzew80oARlcFQNODFOirvzQ== --osd-uuid a90bce8c-2f3b-47d6-8790-977145d7a877 2024-03-29T07:07:26.330 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:26.335+0000 7f99c915c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:07:26.370 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:26.379+0000 7f99c915c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:07:26.372 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:26.379+0000 7f99c915c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:07:26.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:26.379+0000 7f99c915c7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T07:07:26.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:26.379+0000 7f99c915c7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T07:07:26.373 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:26.379+0000 7f99c915c7c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-29T07:07:26.455 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:26.459+0000 7f99c915c7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-29T07:07:33.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-29T07:07:33.573 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-29T07:07:33.574 INFO:tasks.workunit.client.0.smithi123.stdout:adding osd2 key to auth repository 2024-03-29T07:07:33.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-29T07:07:33.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-29T07:07:33.963 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.2 2024-03-29T07:07:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-29T07:07:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:07:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-29T07:07:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-29T07:07:33.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-29T07:07:33.991 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:33.995+0000 7f7f9c0aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:07:33.993 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:33.999+0000 7f7f9c0aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:07:33.995 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:33.999+0000 7f7f9c0aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:07:34.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-29T07:07:34.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:07:34.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-29T07:07:34.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:07:34.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:07:34.303 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:34.303 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:07:34.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:07:34.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:34.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:34.541 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:34.547+0000 7f7f9c0aa7c0 -1 Falling back to public interface 2024-03-29T07:07:34.631 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:35.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:35.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:35.633 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:07:35.633 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:07:35.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:35.634 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:35.951 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:36.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:36.953 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:36.953 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:07:36.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:07:36.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:36.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:37.232 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:07:37.239+0000 7f7f9c0aa7c0 -1 osd.2 0 log_to_monitors true 2024-03-29T07:07:37.261 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:38.262 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:38.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:38.263 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:07:38.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:07:38.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:38.264 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:38.731 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:39.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:39.733 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:39.734 INFO:tasks.workunit.client.0.smithi123.stdout:4 2024-03-29T07:07:39.734 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-29T07:07:39.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:39.735 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:40.054 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:41.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:41.056 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:41.057 INFO:tasks.workunit.client.0.smithi123.stdout:5 2024-03-29T07:07:41.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-29T07:07:41.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:41.057 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:41.470 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-29T07:07:42.472 INFO:tasks.workunit.client.0.smithi123.stdout:6 2024-03-29T07:07:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:42.472 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:42.771 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:43.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:43.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:43.774 INFO:tasks.workunit.client.0.smithi123.stdout:7 2024-03-29T07:07:43.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-29T07:07:43.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:43.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:44.263 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:45.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:45.265 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:45.265 INFO:tasks.workunit.client.0.smithi123.stdout:8 2024-03-29T07:07:45.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-29T07:07:45.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:45.266 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:45.575 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:46.576 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:46.577 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:46.577 INFO:tasks.workunit.client.0.smithi123.stdout:9 2024-03-29T07:07:46.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-29T07:07:46.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:46.578 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:46.994 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:47.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:47.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:47.996 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-29T07:07:47.996 INFO:tasks.workunit.client.0.smithi123.stdout:10 2024-03-29T07:07:47.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:47.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:48.408 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:49.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:49.410 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:49.410 INFO:tasks.workunit.client.0.smithi123.stdout:11 2024-03-29T07:07:49.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-29T07:07:49.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:49.411 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:49.720 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:50.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:50.723 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:50.723 INFO:tasks.workunit.client.0.smithi123.stdout:12 2024-03-29T07:07:50.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-29T07:07:50.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:50.724 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:51.100 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:52.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:52.102 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:52.102 INFO:tasks.workunit.client.0.smithi123.stdout:13 2024-03-29T07:07:52.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-29T07:07:52.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:52.103 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:52.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:53.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:53.914 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:53.915 INFO:tasks.workunit.client.0.smithi123.stdout:14 2024-03-29T07:07:53.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-29T07:07:53.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:53.915 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:54.757 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:55.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:55.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:55.759 INFO:tasks.workunit.client.0.smithi123.stdout:15 2024-03-29T07:07:55.759 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-29T07:07:55.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:55.760 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:56.615 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:57.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:57.617 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:57.617 INFO:tasks.workunit.client.0.smithi123.stdout:16 2024-03-29T07:07:57.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-29T07:07:57.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:57.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:58.185 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:07:59.186 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:07:59.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:07:59.187 INFO:tasks.workunit.client.0.smithi123.stdout:17 2024-03-29T07:07:59.187 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-29T07:07:59.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:07:59.188 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:07:59.908 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:08:00.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:08:00.911 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:08:00.912 INFO:tasks.workunit.client.0.smithi123.stdout:18 2024-03-29T07:08:00.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-29T07:08:00.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:08:00.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:08:01.602 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:08:02.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:08:02.604 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:08:02.604 INFO:tasks.workunit.client.0.smithi123.stdout:19 2024-03-29T07:08:02.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-29T07:08:02.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:08:02.605 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:08:03.239 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:08:04.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:08:04.241 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:08:04.242 INFO:tasks.workunit.client.0.smithi123.stdout:20 2024-03-29T07:08:04.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-29T07:08:04.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:08:04.242 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-29T07:08:04.658 INFO:tasks.workunit.client.0.smithi123.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3935981767,v1:127.0.0.1:6819/3935981767] [v2:127.0.0.1:6820/3935981767,v1:127.0.0.1:6821/3935981767] exists,up a90bce8c-2f3b-47d6-8790-977145d7a877 2024-03-29T07:08:04.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:08:04.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:08:04.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:08:04.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-03-29T07:08:04.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-03-29T07:08:04.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-29T07:08:04.659 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-29T07:08:04.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-29T07:08:04.660 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-29T07:08:05.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-03-29T07:08:05.122 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-03-29T07:08:05.862 INFO:tasks.workunit.client.0.smithi123.stderr:pool 'testpool' created 2024-03-29T07:08:05.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-29T07:08:06.884 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-29T07:08:06.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-29T07:08:06.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:08:06.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:08:06.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:08:06.886 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:08:06.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:08:06.886 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:08:06.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:08:06.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:08:06.887 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:08:07.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:08:07.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:08:07.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:08:07.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:08:07.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:08:07.025 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:08:07.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:08:07.340 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:08:07.340 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T07:08:07.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:08:07.340 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:08:07.341 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:08:07.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-29T07:08:07.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-29T07:08:07.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-29T07:08:07.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:08:07.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:08:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-29T07:08:07.714 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-29T07:08:07.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-03-29T07:08:07.715 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:08:07.715 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T07:08:07.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-29T07:08:07.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-29T07:08:07.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509446' 2024-03-29T07:08:07.881 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:08:07.881 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:08:07.882 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-29T07:08:07.882 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:08:07.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-29T07:08:07.883 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:08:07.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-29T07:08:07.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-29T07:08:07.885 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 21474836502 2024-03-29T07:08:07.885 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:08:08.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-29T07:08:08.220 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:08:08.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-29T07:08:08.221 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:08:08.223 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:08:08.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-29T07:08:08.224 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:08:08.225 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-29T07:08:08.225 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 42949672975 2024-03-29T07:08:08.226 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-29T07:08:08.226 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:08:08.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-29T07:08:08.565 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:08:08.567 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-29T07:08:08.567 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:08:08.569 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T07:08:08.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-29T07:08:08.570 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:08:08.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-29T07:08:08.572 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-29T07:08:08.572 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509446 2024-03-29T07:08:08.573 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:08:08.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509446 2024-03-29T07:08:08.885 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:08:09.887 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:08:09.888 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:08:10.197 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509446 2024-03-29T07:08:10.197 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:08:10.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:08:10.198 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:08:10.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T07:08:10.593 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:08:10.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:08:10.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:08:10.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:08:10.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:08:10.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:08:10.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:08:10.943 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T07:08:10.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:08:10.943 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:08:10.944 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:08:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T07:08:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:08:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:08:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-29T07:08:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-03-29T07:08:11.300 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-03-29T07:08:11.301 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-03-29T07:08:11.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:11.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:11.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:11.488 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:11.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:11.758 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:12.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:12.025 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:12.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:12.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:12.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:12.508 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:12.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:12.779 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:13.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:13.022 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:13.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:13.315 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:13.549 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:13.550 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:13.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:13.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:14.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:14.053 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:14.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:14.270 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:14.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:14.529 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:14.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:14.766 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:15.049 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:15.050 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:15.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:15.275 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:15.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:15.584 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:15.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:15.802 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:16.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:16.031 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:16.281 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:16.282 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:16.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:16.499 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:16.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:16.801 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:17.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:17.035 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:17.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:17.277 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:17.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:17.520 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:17.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:17.738 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:18.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:18.023 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:18.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:18.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:18.556 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-29T07:08:18.557 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-29T07:08:18.786 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-03-29T07:08:18.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-29T07:08:18.787 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-29T07:08:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-29T07:08:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T07:08:18.787 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-29T07:08:18.788 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-29T07:08:18.789 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T07:08:19.101 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-29T07:08:19.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-29T07:08:19.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-29T07:08:19.445 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-03-29T07:08:19.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-03-29T07:08:19.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-29T07:08:19.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-29T07:08:19.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-29T07:08:19.446 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-29T07:08:19.800 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-03-29T07:08:19.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-03-29T07:08:19.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-29T07:08:19.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-29T07:08:19.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-29T07:08:19.801 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-29T07:08:20.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-03-29T07:08:20.191 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-29T07:08:20.191 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:08:20.192 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:08:20.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:08:20.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:08:20.192 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:08:21.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:08:21.506 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-29T07:08:21.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:08:21.506 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:08:21.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:08:21.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:08:21.507 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:08:21.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:08:21.820 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-03-29T07:08:21.821 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:21.822 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-03-29T07:08:21.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:21.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-29T07:08:21.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:21.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:21.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:21.824 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:21.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:21.825 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-29T07:08:23.323 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-03-29T07:08:23.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:23.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-03-29T07:08:23.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:23.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-29T07:08:23.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:23.860 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:23.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:23.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:23.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:23.861 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-29T07:08:25.307 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:104778fc:::obj2:head# 2024-03-29T07:08:25.842 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:25.842 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:25.845 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-29T07:08:26.839 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-03-29T07:08:27.384 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:27.384 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-03-29T07:08:27.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:27.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-29T07:08:27.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:27.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:27.386 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:27.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:27.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:27.387 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-29T07:08:28.367 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-03-29T07:08:28.909 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:28.910 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-03-29T07:08:28.912 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:28.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-29T07:08:28.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:28.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:28.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:28.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:28.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:28.913 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-29T07:08:29.706 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:c52c9666:::obj5:head# 2024-03-29T07:08:30.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:30.234 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:30.236 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-29T07:08:31.356 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:233a42c1:::obj6:head# 2024-03-29T07:08:31.901 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:31.902 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-03-29T07:08:31.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:31.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-29T07:08:31.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:31.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:31.904 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:31.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:31.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:31.905 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-29T07:08:32.766 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:9a09113a:::obj7:head# 2024-03-29T07:08:33.301 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:33.303 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-03-29T07:08:33.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:33.304 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-29T07:08:33.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:33.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:33.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:33.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:33.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:33.305 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-29T07:08:34.126 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:a557efb1:::obj8:head# 2024-03-29T07:08:34.661 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:34.663 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-03-29T07:08:34.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:34.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-29T07:08:34.665 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:34.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:34.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:34.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:34.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:34.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-29T07:08:35.436 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:fe351e27:::obj9:head# 2024-03-29T07:08:35.978 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:35.979 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-03-29T07:08:35.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:35.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-29T07:08:35.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:35.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:35.980 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:35.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:35.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:35.981 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-29T07:08:36.838 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:d1337354:::obj10:head# 2024-03-29T07:08:37.373 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:37.374 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-03-29T07:08:37.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:37.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-29T07:08:37.376 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:37.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:37.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:37.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:37.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:37.377 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-29T07:08:38.182 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-03-29T07:08:38.966 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:38.966 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-03-29T07:08:38.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:38.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-29T07:08:38.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:38.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:38.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:38.969 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:38.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:38.970 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-29T07:08:39.711 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-03-29T07:08:40.498 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:40.499 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:40.501 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-29T07:08:41.231 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-03-29T07:08:41.774 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:41.774 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-03-29T07:08:41.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:41.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-29T07:08:41.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:41.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:41.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:41.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:41.776 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:41.777 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-29T07:08:42.572 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:e0a44829:::obj14:head# 2024-03-29T07:08:43.109 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:43.110 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:43.112 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-29T07:08:43.800 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:ab946124:::obj15:head# 2024-03-29T07:08:44.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:44.339 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-03-29T07:08:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-29T07:08:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:44.341 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:44.342 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-29T07:08:44.994 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-03-29T07:08:45.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:45.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-03-29T07:08:45.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:45.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-29T07:08:45.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:45.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:45.536 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:45.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:45.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:45.537 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-29T07:08:46.329 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-03-29T07:08:46.866 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:46.867 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-03-29T07:08:46.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:46.869 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-29T07:08:46.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:46.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:46.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:46.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:46.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:46.870 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-29T07:08:47.614 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:b9836a99:::obj18:head# 2024-03-29T07:08:48.165 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:48.166 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:48.168 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-29T07:08:48.950 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-03-29T07:08:49.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:49.486 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:49.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-29T07:08:50.268 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:24b5611b:::obj20:head# 2024-03-29T07:08:50.809 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:50.810 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-03-29T07:08:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-03-29T07:08:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:50.812 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:50.813 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-03-29T07:08:51.620 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-03-29T07:08:52.158 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:52.159 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:52.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-03-29T07:08:53.081 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:c0c49def:::obj22:head# 2024-03-29T07:08:53.618 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:53.619 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-03-29T07:08:53.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:53.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-03-29T07:08:53.621 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:53.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:53.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:53.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:53.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:53.622 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-03-29T07:08:54.391 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:242c40e0:::obj23:head# 2024-03-29T07:08:54.926 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:54.927 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:54.929 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-03-29T07:08:55.712 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:7eacede5:::obj24:head# 2024-03-29T07:08:56.245 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:56.246 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-03-29T07:08:56.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-29T07:08:56.248 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-03-29T07:08:56.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:56.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:56.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-29T07:08:56.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:56.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:56.249 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-03-29T07:08:57.156 INFO:tasks.workunit.client.0.smithi123.stdout:remove 0#1:7109478f:::obj25:head# 2024-03-29T07:08:57.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-29T07:08:57.689 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-03-29T07:08:57.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-29T07:08:57.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-03-29T07:08:57.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-29T07:08:57.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-29T07:08:57.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-29T07:08:57.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-29T07:08:57.691 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:08:57.692 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-03-29T07:08:58.474 INFO:tasks.workunit.client.0.smithi123.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-03-29T07:08:59.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:08:59.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:08:59.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:08:59.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-29T07:08:59.013 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:08:59.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-29T07:08:59.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:08:59.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:08:59.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:08:59.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:08:59.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-29T07:08:59.014 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:08:59.015 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:08:59.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:08:59.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:08:59.018 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-29T07:08:59.019 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.1 2024-03-29T07:08:59.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:08:59.020 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-29T07:08:59.020 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-29T07:08:59.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:08:59.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:08:59.021 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:08:59.052 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:08:59.061+0000 7f86550a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:08:59.055 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:08:59.061+0000 7f86550a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:08:59.056 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:08:59.065+0000 7f86550a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:08:59.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-29T07:08:59.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:08:59.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-29T07:08:59.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:08:59.338 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:08:59.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:08:59.339 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:08:59.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:08:59.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:08:59.339 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:08:59.601 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:08:59.609+0000 7f86550a47c0 -1 Falling back to public interface 2024-03-29T07:08:59.666 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:09:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:09:00.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:09:00.668 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:09:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:09:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:09:00.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:09:00.995 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:09:01.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:09:01.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:09:01.997 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:09:01.997 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:09:01.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:09:01.998 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:09:02.220 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:09:02.229+0000 7f86550a47c0 -1 osd.1 23 log_to_monitors true 2024-03-29T07:09:02.353 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:09:03.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:09:03.355 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:09:03.356 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:09:03.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:09:03.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:09:03.356 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-29T07:09:03.681 INFO:tasks.workunit.client.0.smithi123.stdout:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6802/2841555370,v1:127.0.0.1:6803/2841555370] [v2:127.0.0.1:6804/2841555370,v1:127.0.0.1:6805/2841555370] exists,up 85ce611f-a6e8-4fdf-96cf-f4cc812e57b4 2024-03-29T07:09:03.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:09:03.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:09:03.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:09:03.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:09:03.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-29T07:09:03.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-29T07:09:03.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-29T07:09:03.684 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-29T07:09:03.685 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-29T07:09:03.685 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-29T07:09:03.686 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-29T07:09:03.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-29T07:09:03.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-29T07:09:03.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-29T07:09:03.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-29T07:09:03.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-29T07:09:03.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-29T07:09:03.687 INFO:tasks.workunit.client.0.smithi123.stdout:start osd.0 2024-03-29T07:09:03.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-29T07:09:03.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=deb455b1-ea21-4ce9-9654-f092c808f52b --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.61544/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-29T07:09:03.688 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-29T07:09:03.688 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-29T07:09:03.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-29T07:09:03.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-29T07:09:03.689 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-29T07:09:03.720 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:09:03.728+0000 7fc0c74677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:09:03.759 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:09:03.764+0000 7fc0c74677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:09:03.760 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:09:03.768+0000 7fc0c74677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-29T07:09:04.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-29T07:09:04.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-29T07:09:04.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-29T07:09:04.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-29T07:09:04.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-29T07:09:04.046 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:09:04.046 INFO:tasks.workunit.client.0.smithi123.stdout:0 2024-03-29T07:09:04.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-29T07:09:04.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:09:04.047 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:09:04.293 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:09:04.300+0000 7fc0c74677c0 -1 Falling back to public interface 2024-03-29T07:09:04.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:09:05.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:09:05.360 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:09:05.361 INFO:tasks.workunit.client.0.smithi123.stdout:1 2024-03-29T07:09:05.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-29T07:09:05.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:09:05.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:09:05.680 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:09:06.682 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:09:06.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:09:06.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-29T07:09:06.683 INFO:tasks.workunit.client.0.smithi123.stdout:2 2024-03-29T07:09:06.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:09:06.683 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:09:06.964 INFO:tasks.workunit.client.0.smithi123.stderr:2024-03-29T07:09:06.972+0000 7fc0c74677c0 -1 osd.0 21 log_to_monitors true 2024-03-29T07:09:07.001 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-29T07:09:08.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-29T07:09:08.003 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-29T07:09:08.003 INFO:tasks.workunit.client.0.smithi123.stdout:3 2024-03-29T07:09:08.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-29T07:09:08.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-29T07:09:08.004 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-29T07:09:08.413 INFO:tasks.workunit.client.0.smithi123.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/3368899964,v1:127.0.0.1:6811/3368899964] [v2:127.0.0.1:6812/3368899964,v1:127.0.0.1:6813/3368899964] exists,up d9312f8f-1cb2-4093-845c-e6964818107a 2024-03-29T07:09:08.413 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-29T07:09:08.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-29T07:09:08.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-29T07:09:08.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-03-29T07:09:08.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:09:08.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:09:08.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:09:08.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:09:08.415 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:09:08.415 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:09:08.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:09:08.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:09:08.415 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:09:08.544 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:09:08.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:09:08.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:09:08.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:09:08.545 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:09:08.545 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:09:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:09:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:09:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T07:09:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:09:09.019 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:09.019 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:09:09.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-29T07:09:09.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-29T07:09:09.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-29T07:09:09.201 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:09.202 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:09:09.345 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116995 2024-03-29T07:09:09.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116995 2024-03-29T07:09:09.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995' 2024-03-29T07:09:09.346 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:09.346 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T07:09:09.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-29T07:09:09.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-29T07:09:09.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-115964116995 2-64424509459' 2024-03-29T07:09:09.485 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:09.486 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-29T07:09:09.486 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:09.487 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:09:09.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-29T07:09:09.488 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:09.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-29T07:09:09.489 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 128849018882 2024-03-29T07:09:09.489 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-29T07:09:09.489 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:09:09.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-29T07:09:09.773 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:09:10.775 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:09:10.777 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:09:11.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-29T07:09:11.095 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:11.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116995 2024-03-29T07:09:11.096 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:11.098 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:09:11.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116995 2024-03-29T07:09:11.100 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:11.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116995 2024-03-29T07:09:11.101 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116995' 2024-03-29T07:09:11.102 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 115964116995 2024-03-29T07:09:11.102 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:09:11.414 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116995 -lt 115964116995 2024-03-29T07:09:11.415 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:11.416 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-29T07:09:11.416 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:11.418 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T07:09:11.419 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-29T07:09:11.420 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:11.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-29T07:09:11.422 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-29T07:09:11.422 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509459 2024-03-29T07:09:11.423 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:09:11.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-03-29T07:09:11.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:09:11.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:09:11.754 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:09:12.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T07:09:12.163 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:09:12.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:09:12.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:09:12.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:09:12.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:09:12.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:09:12.164 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:09:12.502 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T07:09:12.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:09:12.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:09:12.503 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:09:12.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T07:09:12.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:09:12.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:09:12.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-03-29T07:09:12.857 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-29T07:09:12.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-29T07:09:12.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:09:12.857 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:09:12.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:09:12.858 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:09:13.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:13.227 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-29T07:09:13.515 INFO:tasks.workunit.client.0.smithi123.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-03-29T07:09:13.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:13.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-29T07:09:13.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:13.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-29T07:09:13.532 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-29T07:09:13.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:09:13.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:09:13.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:09:13.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:09:13.533 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:09:13.534 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:09:13.834 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:08:05.722357+0000 '>' 2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:13.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:09:14.835 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:09:14.836 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:09:14.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:09:14.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:09:14.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:09:14.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:09:14.837 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:09:15.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:08:05.722357+0000 '>' 2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:15.154 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:09:16.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:09:16.156 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:09:16.157 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:09:16.157 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:09:16.157 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:09:16.158 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:09:16.158 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:09:16.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:08:05.722357+0000 '>' 2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:16.477 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:09:17.478 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:09:17.479 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:09:17.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:09:17.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:09:17.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:09:17.480 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:09:17.481 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:09:17.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:08:05.722357+0000 '>' 2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:17.797 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-29T07:09:18.798 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-29T07:09:18.799 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-29T07:09:18.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-29T07:09:18.799 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-29T07:09:18.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-29T07:09:18.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-29T07:09:18.800 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-29T07:09:19.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-29T07:09:14.245266+0000 '>' 2024-03-29T07:08:05.722357+0000 2024-03-29T07:09:19.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-29T07:09:19.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-03-29T07:09:19.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-29T07:09:19.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-29T07:09:19.113 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-29T07:09:19.114 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-29T07:09:19.114 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-29T07:09:19.115 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-29T07:09:19.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-29T07:09:19.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-29T07:09:19.115 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-29T07:09:19.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-29T07:09:19.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-29T07:09:19.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-29T07:09:19.232 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-29T07:09:19.233 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:09:19.233 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:09:19.546 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:09:19.547 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:09:19.547 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T07:09:19.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:09:19.547 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:19.547 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:09:19.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-29T07:09:19.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-29T07:09:19.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-03-29T07:09:19.687 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:19.687 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:09:19.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116998 2024-03-29T07:09:19.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116998 2024-03-29T07:09:19.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-115964116998' 2024-03-29T07:09:19.819 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:19.819 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T07:09:19.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-29T07:09:19.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-29T07:09:19.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-115964116998 2-64424509463' 2024-03-29T07:09:19.949 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:19.950 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-03-29T07:09:19.950 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:19.954 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:09:19.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-03-29T07:09:19.954 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:19.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-29T07:09:19.955 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 128849018885 2024-03-29T07:09:19.955 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-03-29T07:09:19.955 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:09:20.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-29T07:09:20.251 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:20.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964116998 2024-03-29T07:09:20.252 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:20.253 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:09:20.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964116998 2024-03-29T07:09:20.255 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:20.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116998 2024-03-29T07:09:20.256 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 115964116998 2024-03-29T07:09:20.256 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964116998' 2024-03-29T07:09:20.257 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:09:20.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116998 -lt 115964116998 2024-03-29T07:09:20.591 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:20.591 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-29T07:09:20.592 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:20.594 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T07:09:20.594 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-29T07:09:20.595 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:20.596 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-29T07:09:20.597 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509463 2024-03-29T07:09:20.597 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-29T07:09:20.597 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:09:20.934 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509463 2024-03-29T07:09:20.935 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:09:21.936 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:09:21.937 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:09:22.259 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-29T07:09:22.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-29T07:09:22.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:09:22.261 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:09:22.649 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-29T07:09:22.650 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-29T07:09:22.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-29T07:09:22.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-29T07:09:22.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-29T07:09:22.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-29T07:09:22.650 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-29T07:09:22.651 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-29T07:09:22.986 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-29T07:09:22.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-29T07:09:22.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-29T07:09:22.987 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-29T07:09:23.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-29T07:09:23.327 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-29T07:09:23.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-29T07:09:23.328 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-03-29T07:09:23.654 INFO:tasks.workunit.client.0.smithi123.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-29T07:09:23.654 INFO:tasks.workunit.client.0.smithi123.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-29T07:09:15.795160+0000 21'30 31:135 [1,0,2] 1 [1,0,2] 1 21'30 2024-03-29T07:09:14.245266+0000 21'30 2024-03-29T07:09:14.245266+0000 0 1 periodic scrub scheduled @ 2024-03-30T07:09:14.245266+0000 17 0 2024-03-29T07:09:23.654 INFO:tasks.workunit.client.0.smithi123.stdout: 2024-03-29T07:09:23.655 INFO:tasks.workunit.client.0.smithi123.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-29T07:09:23.655 INFO:tasks.workunit.client.0.smithi123.stderr:dumped pgs 2024-03-29T07:09:23.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-03-29T07:09:23.670 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-29T07:09:23.670 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-29T07:09:23.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-29T07:09:23.983 INFO:tasks.workunit.client.0.smithi123.stderr:1 2024-03-29T07:09:23.983 INFO:tasks.workunit.client.0.smithi123.stderr:2' 2024-03-29T07:09:23.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-29T07:09:23.983 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:23.984 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-29T07:09:24.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-03-29T07:09:24.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-03-29T07:09:24.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887' 2024-03-29T07:09:24.161 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:24.161 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-29T07:09:24.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117000 2024-03-29T07:09:24.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117000 2024-03-29T07:09:24.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887 1-115964117000' 2024-03-29T07:09:24.296 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-29T07:09:24.296 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-29T07:09:24.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-29T07:09:24.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-29T07:09:24.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018887 1-115964117000 2-64424509464' 2024-03-29T07:09:24.434 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:24.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018887 2024-03-29T07:09:24.435 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:24.435 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-29T07:09:24.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018887 2024-03-29T07:09:24.436 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:24.437 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-03-29T07:09:24.438 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.0 seq 128849018887 2024-03-29T07:09:24.438 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2024-03-29T07:09:24.438 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:09:24.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018886 -lt 128849018887 2024-03-29T07:09:24.750 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-29T07:09:25.752 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-29T07:09:25.753 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-29T07:09:26.062 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-03-29T07:09:26.063 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:26.063 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-115964117000 2024-03-29T07:09:26.063 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:26.065 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-29T07:09:26.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-115964117000 2024-03-29T07:09:26.066 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:26.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117000 2024-03-29T07:09:26.068 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 115964117000' 2024-03-29T07:09:26.068 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.1 seq 115964117000 2024-03-29T07:09:26.068 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-29T07:09:26.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117000 -lt 115964117000 2024-03-29T07:09:26.359 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-29T07:09:26.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-29T07:09:26.360 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-29T07:09:26.361 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-29T07:09:26.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-29T07:09:26.362 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-29T07:09:26.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-29T07:09:26.364 INFO:tasks.workunit.client.0.smithi123.stdout:waiting osd.2 seq 64424509464 2024-03-29T07:09:26.364 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-29T07:09:26.365 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-29T07:09:26.668 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509464 2024-03-29T07:09:26.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-29T07:09:26.669 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_bytes": 210, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects": 30, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_object_clones": 0, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_object_copies": 90, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_missing_on_primary": 0, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_missing": 0, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_degraded": 0, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_misplaced": 0, 2024-03-29T07:09:26.825 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_unfound": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_dirty": 30, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_whiteouts": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_read": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_read_kb": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_write": 30, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_write_kb": 30, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_scrub_errors": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_shallow_scrub_errors": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_deep_scrub_errors": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_recovered": 26, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_bytes_recovered": 182, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_keys_recovered": 0, 2024-03-29T07:09:26.826 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_omap": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_hit_set_archive": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_bytes_hit_set_archive": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush_kb": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict_kb": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_promote": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush_mode_high": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_flush_mode_low": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict_mode_some": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_evict_mode_full": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_pinned": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_legacy_snapsets": 0, 2024-03-29T07:09:26.827 INFO:tasks.workunit.client.0.smithi123.stdout: "num_large_omap_objects": 0, 2024-03-29T07:09:26.828 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_manifest": 0, 2024-03-29T07:09:26.828 INFO:tasks.workunit.client.0.smithi123.stdout: "num_omap_bytes": 0, 2024-03-29T07:09:26.828 INFO:tasks.workunit.client.0.smithi123.stdout: "num_omap_keys": 0, 2024-03-29T07:09:26.828 INFO:tasks.workunit.client.0.smithi123.stdout: "num_objects_repaired": 26 2024-03-29T07:09:26.828 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:09:26.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-29T07:09:26.828 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-03-29T07:09:26.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-03-29T07:09:26.961 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-03-29T07:09:26.962 INFO:tasks.workunit.client.0.smithi123.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-03-29T07:09:26.962 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-03-29T07:09:26.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-29T07:09:26.963 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-03-29T07:09:26.964 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-03-29T07:09:26.964 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-03-29T07:09:26.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-29T07:09:26.965 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-29T07:09:27.223 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout:{ 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "osd": 0, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "up_from": 30, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "seq": 128849018887, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "num_pgs": 1, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "num_osds": 1, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_osds": 1, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "num_per_pool_omap_osds": 1, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "kb": 104857600, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used": 28480, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_data": 376, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_omap": 4, 2024-03-29T07:09:27.239 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_used_meta": 28091, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "kb_avail": 104829120, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "statfs": { 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "total": 107374182400, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "available": 107345018880, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "internally_reserved": 0, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "allocated": 385024, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "data_stored": 206078, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed": 0, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_allocated": 0, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "data_compressed_original": 0, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "omap_allocated": 4402, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "internal_metadata": 28765902 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: "hb_peers": [ 2024-03-29T07:09:27.240 INFO:tasks.workunit.client.0.smithi123.stdout: 1, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: 2 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: ], 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "snap_trim_queue_len": 0, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "num_snap_trimming": 0, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "num_shards_repaired": 26, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "op_queue_age_hist": { 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "histogram": [], 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "upper_bound": 1 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "perf_stat": { 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ms": 0, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ms": 0, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "commit_latency_ns": 0, 2024-03-29T07:09:27.241 INFO:tasks.workunit.client.0.smithi123.stdout: "apply_latency_ns": 0 2024-03-29T07:09:27.242 INFO:tasks.workunit.client.0.smithi123.stdout: }, 2024-03-29T07:09:27.242 INFO:tasks.workunit.client.0.smithi123.stdout: "alerts": [] 2024-03-29T07:09:27.242 INFO:tasks.workunit.client.0.smithi123.stdout:} 2024-03-29T07:09:27.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-29T07:09:27.242 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-29T07:09:27.516 INFO:tasks.workunit.client.0.smithi123.stderr:dumped all 2024-03-29T07:09:27.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-03-29T07:09:27.533 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-03-29T07:09:27.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-03-29T07:09:27.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-03-29T07:09:27.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-03-29T07:09:27.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-03-29T07:09:27.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-29T07:09:27.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-03-29T07:09:27.534 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-29T07:09:27.535 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-29T07:09:27.535 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-29T07:09:27.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-29T07:09:27.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-29T07:09:27.535 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-29T07:09:27.846 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-29T07:09:27.846 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-29T07:09:27.848 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-29T07:09:27.848 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-29T07:09:27.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-29T07:09:27.850 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-29T07:09:27.851 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-29T07:09:27.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:09:27.852 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-29T07:09:27.853 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-29T07:09:27.853 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T07:09:27.855 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:09:27.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-03-29T07:09:27.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-03-29T07:09:27.856 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-03-29T07:09:27.858 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-03-29T07:09:27.859 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-29T07:09:27.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-29T07:09:27.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-29T07:09:27.873 INFO:tasks.workunit.client.0.smithi123.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.61544 2024-03-29T07:09:27.873 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.61544 2024-03-29T07:09:27.874 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-29T07:09:27.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-29T07:09:27.875 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-03-29T07:09:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T07:09:27.876 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-03-29T07:09:27.877 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-29T07:09:28.214 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-29T07:09:28.471 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c98218619216499f9f372ce10d1e3e6f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_8da23624bc17fe0b1f797227e72a0db0ce7821ba/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-29T07:09:28.474 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T07:09:28.484 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T07:09:28.485 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T07:09:28.527 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T07:09:28.537 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T07:09:28.537 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T07:09:28.558 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2024-03-29T07:09:28.559 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2024-03-29T07:09:28.559 INFO:teuthology.orchestra.run.smithi123.stdout:*hv01.front.sepi 44.4.53.2 2 u 29 256 377 0.188 -1.940 0.372 2024-03-29T07:09:28.559 INFO:teuthology.orchestra.run.smithi123.stdout:+hv02.front.sepi 63.231.80.2 3 u 89 256 377 0.199 -1.293 0.363 2024-03-29T07:09:28.559 INFO:teuthology.orchestra.run.smithi123.stdout:+hv03.front.sepi 74.6.168.72 3 u 69 256 377 0.158 -4.294 0.931 2024-03-29T07:09:28.559 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-29T07:09:28.560 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T07:09:28.570 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T07:09:28.570 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T07:09:28.591 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T07:09:28.612 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T07:09:28.626 INFO:teuthology.task.internal:Duration was 6820.129618 seconds 2024-03-29T07:09:28.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T07:09:28.644 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T07:09:28.645 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T07:09:28.683 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T07:09:28.683 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-03-29T07:09:28.683 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T07:09:28.733 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T07:09:28.733 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T07:09:28.816 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T07:09:28.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T07:09:28.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T07:09:28.923 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T07:09:28.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T07:09:28.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T07:09:28.948 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:09:28.989 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-03-29T07:09:29.017 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T07:09:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T07:09:29.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T07:09:29.078 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T07:09:29.079 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi/7628612/remote/smithi123 2024-03-29T07:09:29.080 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T07:09:30.062 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T07:09:30.062 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T07:09:30.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T07:09:30.119 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T07:09:30.119 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T07:09:30.137 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T07:09:30.137 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T07:09:30.156 INFO:teuthology.orchestra.run.smithi123.stdout: 658624 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 29 07:09 /home/ubuntu/cephtest 2024-03-29T07:09:30.157 INFO:teuthology.orchestra.run.smithi123.stdout: 671694 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 29 05:24 /home/ubuntu/cephtest/mnt.0 2024-03-29T07:09:30.157 INFO:teuthology.orchestra.run.smithi123.stdout: 673721 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 29 05:36 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-29T07:09:30.158 INFO:teuthology.orchestra.run.smithi123.stdout: 686465 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 29 05:36 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-29T07:09:30.158 INFO:teuthology.orchestra.run.smithi123.stdout: 686473 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 29 07:09 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-03-29T07:09:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T07:09:30.159 INFO:teuthology.orchestra.run.smithi123.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-29T07:09:30.159 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-29T07:09:30.167 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T07:09:30.179 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T07:09:30.191 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi123 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-29T07:09:30.191 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T07:09:30.219 DEBUG:teuthology.nuke:shortname: smithi123 2024-03-29T07:09:30.220 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T07:09:30.244 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-28_21:00:03-rados-squid-distro-default-smithi/7628612', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 05:07:20.671585', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL5LjwnX1CVfZXJX4VTbgv2L4/bti+oveyIlQz+3mHCwvt/qq8l19OdTSDuMkMpl8bKLAYTRDBRvSmfI9cuHMyk='} 2024-03-29T07:09:30.297 INFO:teuthology.orchestra.console.smithi123:Power off 2024-03-29T07:09:30.298 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T07:09:30.324 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-03-29T07:09:30.325 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T07:09:30.351 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-29T07:09:34.353 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T07:09:34.464 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-29T07:09:38.466 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T07:09:38.577 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-29T07:09:42.578 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T07:09:42.689 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-03-29T07:09:42.789 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-03-29T07:09:42.895 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 6820.129618167877 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi123 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8da23624bc17fe0b1f797227e72a0db0ce7821ba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c98218619216499f9f372ce10d1e3e6f status: fail success: false 2024-03-29T07:09:42.895 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T07:09:42.981 INFO:teuthology.run:FAIL